{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "EFyh_Kfx5jkw" }, "source": [ "**Chapter 15 – Processing Sequences Using RNNs and CNNs**\n", "\n", "**Chapter 16 – Natural Language Processing with RNNs and Attention**" ] }, { "cell_type": "markdown", "metadata": { "id": "TQs1xzgo5jkz" }, "source": [ "_This notebook contains the sample from https://github.com/ageron/handson-ml2/ and https://github.com/fchollet/deep-learning-with-python-notebooks_" ] }, { "cell_type": "markdown", "metadata": { "id": "qIV46eeo5jkz" }, "source": [ "\n", " \n", " \n", "
\n", " \"Open\n", " \n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "PdY3OIJ15jk0" }, "source": [ "# Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "ykB4PPYw5jk0" }, "source": [ "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0cmPr4J_5jk1" }, "outputs": [], "source": [ "# Python ≥3.5 is required\n", "import sys\n", "assert sys.version_info >= (3, 5)\n", "\n", "# Is this notebook running on Colab or Kaggle?\n", "IS_COLAB = \"google.colab\" in sys.modules\n", "IS_KAGGLE = \"kaggle_secrets\" in sys.modules\n", "\n", "# Scikit-Learn ≥0.20 is required\n", "import sklearn\n", "assert sklearn.__version__ >= \"0.20\"\n", "\n", "# TensorFlow ≥2.0 is required\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "assert tf.__version__ >= \"2.0\"\n", "\n", "if not tf.config.list_physical_devices('GPU'):\n", " print(\"No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\")\n", " if IS_COLAB:\n", " print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n", " if IS_KAGGLE:\n", " print(\"Go to Settings > Accelerator and select GPU.\")\n", "\n", "# Common imports\n", "import numpy as np\n", "import os\n", "from pathlib import Path\n", "\n", "# to make this notebook's output stable across runs\n", "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "# To plot pretty figures\n", "%matplotlib inline\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "mpl.rc('axes', labelsize=14)\n", "mpl.rc('xtick', labelsize=12)\n", "mpl.rc('ytick', labelsize=12)\n", "\n", "# Where to save the figures\n", "PROJECT_ROOT_DIR = \".\"\n", "CHAPTER_ID = \"rnn\"\n", "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n", "os.makedirs(IMAGES_PATH, exist_ok=True)\n", "\n", "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n", " path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n", " print(\"Saving figure\", fig_id)\n", " if tight_layout:\n", " plt.tight_layout()\n", " plt.savefig(path, format=fig_extension, dpi=resolution)" ] }, { "cell_type": "markdown", "metadata": { "id": "eAE6jniY5jk2" }, "source": [ "# Basic RNNs for forecasting times series" ] }, { "cell_type": "markdown", "metadata": { "id": "PhK1Eyvw5jk3" }, "source": [ "## Generate the Dataset" ] }, { "cell_type": "markdown", "source": [ "Suppose you are studying the number of active users per hour on your website, or the daily temperature in your city, or your company’s financial health, measured quarterly using multiple metrics. In all these cases, the data will be a sequence of one or more values per time step. This is called a time series. In the first two examples there is a single value per time step, so these are **univariate time series**, while in the financial example there are multiple values per time step (e.g., the company’s revenue, debt, and so on), so it is a **multivariate time series**. A typical task is to predict future values, which is called forecasting. Another common task is to fill in the blanks: to predict (or rather “postdict”) missing values from the past. This is called imputation. \n", "\n", "For simplicity, we are using a time series generated by the generate_time_series() function, shown here:" ], "metadata": { "id": "ZNbWWTJm9_WN" } }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "4XG7SRs35jk3" }, "outputs": [], "source": [ "def generate_time_series(batch_size, n_steps):\n", " freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1)\n", " time = np.linspace(0, 1, n_steps)\n", " series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10)) # wave 1\n", " series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20)) # + wave 2\n", " series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5) # + noise\n", " return series[..., np.newaxis].astype(np.float32)" ] }, { "cell_type": "markdown", "source": [ "The function returns a NumPy array of shape `[batch size, time steps, 1]`, where each series is the sum of two sine waves of fixed amplitudes but random frequencies and phases, plus a bit of noise. \n", "\n", "When dealing with time series (and other types of sequences such as sentences), the input features are generally represented as 3D arrays of shape `[batch size, time steps, dimensionality]`,where dimensionality is 1 for univariate time series and more for multivariate\n", "time series.\n" ], "metadata": { "id": "3mT-BlwO-QMs" } }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "jGFqAitJ5jk4" }, "outputs": [], "source": [ "np.random.seed(42)\n", "\n", "n_steps = 50\n", "series = generate_time_series(10000, n_steps + 1)\n", "X_train, y_train = series[:7000, :n_steps], series[:7000, -1]\n", "X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1]\n", "X_test, y_test = series[9000:, :n_steps], series[9000:, -1]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "UXAjbgOt5jk4", "outputId": "5de60e8b-2b38-47ac-e541-a42fcd6ba328", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "((7000, 50, 1), (7000, 1))" ] }, "metadata": {}, "execution_count": 5 } ], "source": [ "X_train.shape, y_train.shape" ] }, { "cell_type": "markdown", "source": [ "`X_train` contains 7,000 time series (i.e., its shape is `[7000, 50, 1]`), while `X_valid` contains 2,000 (from the 7,000th time series to the 8,999th) and X_test contains 1,000 (from the 9,000 to the 9,999 ). **Since we want to\n", "forecast a single value for each series**, the targets are column vectors (e.g., `y_train` has a shape of `[7000, 1]`)." ], "metadata": { "id": "LXt8l_8F-2Ed" } }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "_LQXcP-r5jk4", "outputId": "27d1aa45-0690-437f-b77a-ae48446f7b66", "colab": { "base_uri": "https://localhost:8080/", "height": 314 } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Saving figure time_series_plot\n" ] }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1sAAAEYCAYAAACqb+zuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXhb1bXw/++WZNmW5zFO7NiOnTkhgTgQlzAEKKGUsQXa0gAXuEBvW0pbytBbWuDyK3SA8pa+b4FCS4fb0JZShhJKoTQxYYgzmIwmtuM4dhw7nkd5ki3t3x+SjG3sxINmrc/z6El0dHS0jyUdnXX22msrrTVCCCGEEEIIITzL4O8GCCGEEEIIIUQokmBLCCGEEEIIIbxAgi0hhBBCCCGE8AIJtoQQQgghhBDCCyTYEkIIIYQQQggvkGBLCCGEEEIIIbxAgi0hhBBCCCGE8IKADLaUUrcrpXYppQaUUr87ybrfVko1KKW6lFLPKaUiRzyWq5TaopTqVUqVKaU+7fXGCyGEEEIIIQQBGmwB9cAPgedOtJJS6iLgu8AFQA6QB/zPiFX+BOwGUoD7gBeVUmneaLAQQgghhBBCjKS01v5uw4SUUj8EsrTWN07w+PNAtdb6e677FwAbtdYZSqmFwH4gVWvd7Xr8XdfjT/tkB4QQQgghhBBhy+TvBszQMuDVEff3ArOUUimux6rcgdaIx5eNtyGl1G3AbQBRUVEF2dnZ3mmxjzgcDgyGQO24nJxQ2AcIjf0I1X2oqKho0VpPqrdbjhGBJxT2AUJjP0J1H+QYEXrvabAJhX2A0NiP6R4jgj3YigU6R9x3/z9unMfcj2eOtyGt9TPAMwCLFi3S5eXlnm2pjxUVFbFu3Tp/N2NGQmEfIDT2I1T3QSlVM9nnyzEi8ITCPkBo7Eeo7oMcI9b5uxkzIvsQOEJhP6Z7jAjuEBOsQPyI++7/d4/zmPvxboQQQgghhBDCy4I92CoFVo64vxJo1Fq3uh7LU0rFjXm81IftE0IIIYQQQoSpgAy2lFImpVQUYASMSqkopdR4KY9/AP5TKbVUKZUIfB/4HYDWugLYAzzgev7ngBXA33yyE0IIIYQQQoiwFpDBFs6gqQ9nWffrXP//vlIqWyllVUplA2it/wn8FNgCHAVqgAdGbOdLwGqgHfgxcLXWutlneyGEEEIIIYQIWwFZIENr/SDw4AQPx45Z93Hg8Qm2Uw2s81zLhBBCCCGEEGJyAjLYCnRdXV00NTUxODjo76ZMKCEhgYMHD/q7GTOSkJBAZWUl6enpxMePrXUihBBCCCFEYJNga4q6urpobGwkMzOT6OholFL+btK4uru7iYuLO/mKAayrqwuTyURdXR2ABFxCCCGEECKoBOqYrYDV1NREZmYmFoslYAOtUKGUwmKxkJmZSVNTk7+bI4QQQgghxJRIsDVFg4ODREdH+7sZYSU6OjqgUzaFEEIIIYQYjwRb0yA9Wr4lf28hhBBCCBGMJNgSQgghhBBCCC+QYEsIIYQQQgghvECCLSGEEEIIIYTwAgm2woBS6oS3G2+80W9ty83N5bHHHvPb6wshhBBCCOEtEmz5ycaNkJsLBoPz340bvfdax48fH749++yzn1j2xBNPTGl7NpvNG80UQgghhBAipEiw5QcbN8Jtt0FNDWjt/Pe227wXcGVkZAzfEhMTRy3r6enhhhtuICMjg5iYGFatWsWmTZtGPT83N5cHH3yQm2++mcTERDZs2ADAc889R3Z2NhaLhcsuu4wnn3zyE5UDX3vtNQoKCoiKimLevHncd999w8HaunXrqKmp4e677x7uZRNCCCGEECJUSLDlB/fdB729o5f19jqX+5rVauXiiy/mX//6F3v37uWqq67i85//PGVlZaPWe/zxx1m8eDG7du3ikUceYdu2bdxyyy18/etfZ8+ePVx++eU88MADo57z5ptvsmHDBm6//XZKS0t57rnnePHFF/ne974HwEsvvURWVhb333//cC+bEEIIIYQQocLk7waEo6NHp7bcm1auXMnKlSuH799333289tprvPjii3z/+98fXn7uuedyzz33DN+///77Wb9+Pffeey8ACxcuZOfOncNpigAPP/wwd999NzfddBMA+fn5/OQnP+G6667j0UcfJTk5GaPRSFxcHBkZGd7eVSGEEEIIIXxKerb8IDt7asu9qaenh3vuuYelS5eSlJREbGwsu3bt4uiYyG/16tWj7peVlXHGGWeMWrZmzZpR90tKSnj44YeJjY0dvn35y1+mp6eHhoYG7+yQEEIIIYQQAUJ6tvzg4YedY7RGphJaLM7lvnbXXXfxz3/+k8cee4wFCxZgsVi44YYbPlEEIyYmZsrbdjgcPPDAA1xzzTWfeCwtLW3abRZCCCGEECIYSLDlB676Etx3nzN1MDvbGWi5l/vSe++9xw033MBVV10FQH9/P4cPH2bhwoUnfN7ixYvZuXPnqGU7duwYdX/VqlWUlZUxf/78CbdjNpux2+3TbL0QQgghhBCBS4ItP9mwwT/B1VgLFy7k5Zdf5oorriAiIoL/+Z//ob+//6TPu+OOOzjrrLN49NFHufLKK9m6dSsvv/zyqHXuv/9+Lr30UnJycvjCF76AyWTiwIED7Nixg5/+9KeAs9Lhu+++y3XXXUdkZCSpqale2U8hhBBCCCF8TcZshbnHH3+c9PR0zj77bC6++GIKCws5++yzT/q8T33qUzz77LP84he/YMWKFbzyyivce++9REVFDa9z0UUX8frrr7NlyxbOOOMMzjjjDH784x+TPWJw2kMPPURtbS35+fmSWiiEEEIIIUJKwPZsKaWSgd8A64EW4L+11s+Ps94bwMjowAyUa61PcT1eDcwC3LlqH2it13ux6QHt6quvRms9fD8nJ4e333571Dp33XXXqPvV1dXjbuvmm2/m5ptvHr7/7W9/+xMpg+vXr2f9+on/3IWFhezdu3eyzRdCCCGEECJoBGywBfwSsOEMlE4FXldK7dVal45cSWt98cj7SqkiYPOYbV2mtX4b4VGPPvooF154IbGxsbz99ts8/fTTPPLII/5ulhBCCCGEEAEhIIMtpVQMcBWwXGttBd5TSv0duB747gmel4uzl+tG77dS7Nq1i8cee4zOzk7mzZvHj370I775zW/6u1lCCCGEEGFtx5FWtla0cN7idApykvzdnLCmRqaUBQql1GnA+1pry4hldwHnaq0vO8Hz7gfO11qvG7GsGojGOT5tN3C31voTeWtKqduA2wDS0tIKXnjhhXFfIyEh4YTV9QKF3W7HaDT6uxkzMnIfKisrKaluo6zNzuJkI/OTgmffrFYrsbGx/m7GjITqPpx33nklWuvVEzxllMkeI4JFqL6nwSgU9iNU90GOEaH3ngab6exDZbudR3b049AQYYB7T4/y+3lTqL4XkzlGBGTPFhALdI1Z1gnEneR5NwA/HLNsA/AhoIBvAm8qpRZrrTtGrqS1fgZ4BmDRokV63bp1477AwYMHiYs7WTP8r7u7OyjaeSIj98FgMvNoiQ3bkIPICDsbbykMmis1RUVFTPR5ChayD5M/RgQLeU8Dhzf3o6SmneKqVgrzUrx6zAyF90KOEaPJexoYprMPpVsqcehyAIYcMJCYw7p1/u0oCNf3AgK3GqEViB+zLB7onugJSqmzgAzgxZHLtdbva637tNa9WusfAR2MLqgxZYHYGxjKtNb0D9kZGHKgAduQg+KqVn83Swi/K6lu45dbKimpafd3U0QAKqlp59pni/nZW+Vs+HWxfE6ECBPzUmKG/68UFOal+LE1IlCDrQrApJRaMGLZSqB0gvUB/gN4yTXG60Q0zl6uaYmIiKCvr2+6TxfT0NfXhznCPGqZHDhEONNa8/DrH3HV09t47E05kRbje6eiCduQA4eGQblIJUTYaOkZAGDZ7HgcGtLjIv3couC2cSPk5sL5559Lbq7z/lQEZLClte4BXgIeUkrFKKXWAlcA/zve+kqpaOALwO/GLM9WSq1VSpmVUlFKqbuBVOD96bYtPT2duro6ent7pYfLy7TW9Pb2UldXR/Og80CRlRSNQ0NqrPkkzxYiNPUMDHHHn/fw7LtHAKS3V0yoqrln+P9KKblIJUSYKCpvJifFwjP/sRql4KUP6/zdpKC1cSPcdhvU1IDWipoa5/2pBFyBOmYL4GvAc0AT0Ap8VWtdqpQ6G3hDaz1yhNqVONMDt4zZRhzwFJAP9AN7gIu11tM+K4mPd2Y31tfXMzg4ON3NeF1/f/+oCYaDUX9/P3FxccyaNYuHth4kMzGav/7Xpzj7J1v4w7YafnDpUn83UQifKalp5/V99bxZ2sDxzn6+vCabv5UcY8DVczE/PbgHHgvPqmjs5o0DDVy4JJ3DzT00WwdYOntsdr4QItT0D9r54HALXzo9m8zEaM7MT+HFD2v5xvnzMRimndgVtu67D3p7Ry/r7XUu37BhctsI2GBLa92GM4gau/xdnAU0Ri77E/CncdYtBVZ4um3x8fHDQVegKioq4rTTTvN3M2bEvQ/H2nt5r7KFb16wgNkJ0XxmeQYv7KrlzgsXEhMZsB9hITympKadLz2zjUG7szf9wcuWcuPaeVy1KovX99fz5+21PPH2Ic5ekIrFLN+JcKe15sG/lxJjNvLjq1ZwpKWHq5/exp93HuWmtfP83TwhhBcVV7XSP+hg3aI0AK4uyOLbf9nLzuo21kjv9pQdPTq15eMJyDRCIUb6W4mz+/vqgiwAbjwzl+7+IV7eLd3iIjz866OG4UDLqKDHZgegICeJ+y9dxi+vW0VZQxffeWEvDoekN4e7Nw408MHhVr6zfhEpsZGszk3mjNxkntlahW3I4e/mCSG8qKi8mUiTYTht+KJlGcRGmnix5JifWxacsrOntnw8EmyJgOZwaP5aUsva/FSykpzTrhXkJLE8M57ff1At4+ZEWIgwOg/VBgURI35E3c5blM73PruENw40cN1vtkuxjDDWZ7Pzw00fsTgjjg1rPj4b+Op5+Rzv7OeVPXKRSgSuzWWNPP5W+ahjWElNu1RdnYKi8ibOzE8hKsI5r5bFbOKSU2bz+v7j9AwM+bl1wefhh8FiGb3MYnEunywJtkRA21bVyrH2Pr5w+tzhZUop/uNTuRxqsvLBYSkKIEJfY1c/sZFG7rxw4YRzzJ02NxGjUnxwuJVrn5XqhOHqyaJK6jv7+Z/Ll2EyfvwTv25hGktnx/P0O4exS++nCEBby5u5+Xe7+MXmSq566gM++8RWbv3DLr74q20yfcEkHWnpobq1l/MWp49afvXqLHptdt440OCnlgWvDRvgmWcgJweU0uTkOO9PdrwWSLAlAtxfdtYSH2Vi/dJZo5ZftnIOyTFmfvdBtX8aJoQPbT/Sxpn5qdx+/oIJJ6YtPtKGxnkSLWW+w9M/9h3nyaLDnDU/9RNjM5RSfO28fKqae3izVE64ROD5Q3H18P8V0NE3yNaKZoYcWqYvmKQtZU0ArFs4OthanZNEboqFF0tq/dGsoDO2N3XDBqiuhs2b36G6emqBFkiwJQJYz6Dmn6UNXHla5nB3uFtUhJEvnT6Xfx9spLatd4ItCBH8jnf2UdPae9KBzYV5KZhNzkO6TGIZft4qbeDrz3+I3aHZWd02bg/AxctnMy81hieLKiUFWwScIy09KJzjUiMjDPzfa1fx/C1rAGfwNV4KtRitqKKZvLQYslNG570ppbi6IIviqjY5ZzqJkpp2rnn6A4/OYSnBlghYxceHsA05+MLqueM+fl1hDgDfeWGPpBaIkLW9qg2ANfOST7heQU4SG28pZGVWApEmIyuzEnzRPBEAyhq6uPOFvbjDpyH7+D0ARoPiK+fkcaCui3te3CfHTREwyhu6Odzcw41rc7lz/aLhdOmC3GRyki3MS4uZMIVaOPXahiiuauW8RenjPv65Vc4iY/f+Tb77J7K5rBGHds5h6aneVAm2REAqqWnntcOD5KZYWJ45/knj8c5+AHZUt/NlGaMiQtT2I63ERZlYMok5kgpykrj5rHn0Ddo5eLzbB60T/vZ+ZQvXPLWNCKPCbDJgnKCIiltuqvOK919LjskYGBEwXiypxWRQ3H7efL5+3vxRQdXi2XEokEDrJLYdbsU25Jgw2Gro7Meg4IPDrfLdP4FEixmYuCDVdMiELCLglNQ4g6eBIY11sI+SmvZxD7IjrzYMDDkoKm+Sg7EIOdur2jgjNxnjJCejdP8wbD/SyinSuxWySmraee69Kv5Z2kB+Wiy/u+kMjnf2U1zVSmFeyoTHwpKajuH/u6/aynFT+NOg3cHLu+u4YEk6KbGRn3g8JyWGLeXNOBxaJuU9gaLyZixmI6fPm2Bcb1Ur7uxh+e5PrM81tco3zp/POQvTPfI3kp4tEXCKq1qH54LRWk/Yheseo+I+9r59sJGBIbuvmimE1zV19VPV0sOavBOnEI40Kz6K3BQLxa70QxF63JNcv76/Aa3h+5csZU5iNAU5SZ/oFRirMC8Fk+ugaTLKGBjhf0XlzbRYbVxTMP6QgexkC7YhBw1d/T5uWfDQWrOlvIkz81OJNBnHXacwLwWT0fndN8p3f0L76zrJS43h2xcu8lgwKsGWCDiFeSkoVwB1oi5c9xiV76xfxLc+vYCDx7u558V9MvA7jPyrtJH/86/ykE2H2H7EPV5raj+Ka+alsLO6TSY4DlHFVa3Dk1wrnCcHk1WQk8Qjn1sOwO0nCcyE8IW/7qolNdbMuYvSxn08x1XsoaZVCjtM5LW99Rxr7yM/LWbCdQpykvjpVSsB+Nq6fPnuT+BAXafHs0IkjVAEnIKcJJJjzEQxyBPXn3hAbEFO0vDjEUYDj75ZToRRMS819oSpNCL4lNS0U1zVyuKMOI6197Fxew0VjVYAniw6zJ9vLaQgd/I9QMGguKqV2EgTy+acfLzWSGvykvnLrlrKGrpZOsXnisDnLpYy3QptV56WxfdfKcUqE5wKP2u1DrC5rImb1uYOT94+Vk6yM4A42tbDp/KlN2askpp27nxhLwC/+6Ca9csyJjz3+eyKDO56UclcexNo7h7geGc/p0xQK2C6JNgSAaepu58Wq40vLTJPKVj62rp8dh9t58WSOhTO0rFSvSg0lNS0c+2zxcPppQBpsWYUropBds2dL+zlD/95BjkpE1/ZCzbbj7SxOjdp1OS0k3GG62R8+5FWCbZCUGyU86f74uUZ/OfZeVM+xplNBpbMjmPvsY6Tryympa1fTzjeWHzslT31DDk010xQdRhgTmIUJoOSnq0JFFe1MuQKntyVSCf63EWajOSkWKhssvqyiUHjgCtLwNPBlqQRioCz56jzBCA/cWofT6UUK+cmAp4t2Sn8b+Q4PgXcdGYuT1+/msgIZ/U1k0HR1D3A+v+zle+/fID/u/lQ0KcWtlgHqGyyTjmFECAryUJmYvRw2XgRWtzv6/cuWTLtk/kVWYkcqOuSVFMv6bJpqfh2Elpr/rqrlpVZCSycFTfheiajgcykaGpkfqhxFeal4C4bMpme7vlpsRySYGtc++s6UQqWSbAlQt3u2g5MBkVO/NQ/nmfmp8rg7xC0wpU/7e6xvHTlnOExe3euX8RfvvIptty1jlPnJvDH7TX87K2KoD/R2eEerzWF4hgjrclLZkd1m4xhDEHbj7SSmRhNVpLl5CtPYEVWAtaBIapa5KTLW2xywe+ESuu7KGvo5uqCrJOum51s4aj0bI1rXmoMGjh3YdqksnkWzIqluqWHQbvjhOuFo/11ncxLjSE20rOJfxJsiYCz52gHi2fHYTZOvcRrQU4S9312CQD/ffFiSeEIMVevzhr1YzKy+lpGQhTnLEwfvsIX7D2b26tasZiN005nKJyXQluPTa5ghhitNTuOtJ10kuuTcWcB7K2dfHENMTVGg5ILfifwZFElRoMaHpN1IjkpFqpbe+Ti0Tjc6cD/de7kil4sSI9jyKGpae3xdtOCzv5jnazwcK8WSLAlAozdodl3rINTXScC03HxKbMBZD6OEPJhTQdKwf2XLj1pWWt3aqGnJiP0l+1H2ijISZpw0PjJuHvEtgdxwCk+6XCzlRarbdo9nm75abFYzEb2ybgtr7l4+cSFCsLd9qpW/rG/AbtDc9sfd500CyE3JYbu/iE6egd91MLgsbfW+fs42Qp689NjATjUKBfiRmruHqChq5/lEmyJUFfZZKXHZufUudP/gZoVH0lspInDckU/ZJQcbWfRrDjioiJOuN7I1MJgLo7S1mOjrKF7Rr0X2ckWMuKjKD4i47ZCyXSnAxjLaFAsz0xg7zHp2fKGSCM0dA34uxkB64VdtcP/n0wWQnayq/y7jNv6hL21HSxIj5106lt+WixKIVkPY3irOAZIsCUCzJ5a59WtmfRsKaXIT4uhslkOJKHA4dDsPtrOadmTC5wmM7FroPt4vNb0T6iVUqzJS2Z7lYzbCiXbq9qYFR85PPfQTKzMSuCj412jqnwKzzAbFQfqOqXE9gSau52B6GSzENxVZiX1bTStNXuPdbIya/LnTNFmI1lJ0RJsjbHvmHeKY0AAB1tKqWSl1MtKqR6lVI1S6ssTrPegUmpQKWUdccsb8fipSqkSpVSv699TfbcXYqr21HYQF2UiL3Vm5bvz02OltGmIqGy20t0/FNTB01S9trcOk0Fhd8zsJHjNvBRarANUtcgJSijQWrP9SCtnzEtBqZmnSa/ISsQ25KCisdsDrRMjRRqh12bnsFz0+4Q+m51dNe18ekn6pLMQ3D1bUiRjtGPtfbT12IbHYE7W/DQ5Rxprf10neV4ojgEBHGwBvwRswCxgA/CUUmrZBOv+RWsdO+JWBaCUMgOvAn8EkoDfA6+6losAtPuoc7zWTMdbzU+PpbFrgK5+ye8Odh+6cvlXZU+/tzOYlNS084/9DQw5NDf+dueMKip+PG5LUglDQU1rL41dAzMujuHmvhou8215XqSrwNPeWvnbjvXvskZ6bXZuXjtv0lkI0WYj6XGRkkY4hvu7O9VsoAWz4jjcbJWe1xEO1HV6JYUQAjTYUkrFAFcBP9BaW7XW7wF/B66f4qbW4Zy4+eda6wGt9S9wVo8+35PtFZ7RMzBERWM3p80ghdBtfppzAKiM2wp+JTXtJFkimDfD3s5gsbWiGffP30wrKualxpAaG8n2I1IkIxS400sLZ1gcw21ucjRJlgj2SUVCj4swQGykiX0yJu4T/r6nnvS4yCmnSeekSPn3sfbWdmA2GViUMfE8ZeOZnx6LbchBrQSvADR199PQ1c8pU0jHnArP95V5xkJgSGtdMWLZXuDcCda/TCnVBhwH/p/W+inX8mXAPj16wMI+1/J/jtyAUuo24DaAtLQ0ioqKZrwT/mS1WoNuH8ra7Dg0GDpqKSo6PqN9aOtxpl9t2rqLzqwTF1XwtmB8L8by5z68V9ZLdoyBd955Z0bbmek++OoY0Vbv7I1VOMczRHbUUFR0bNrbSzMP8ub+ep6NbGVB0seHfPlcBo7J7ser+waIN0Nt6S6OfeSZaquZFgcflNdRVDSz3s9QeC88fYxYFePgvYO1FCW2eKiFvuWN97RnULP5YC/nZ5t4d+vUjulm2wClrfYptSnUP5fvHOhjbgy8/+7WKW2zq8MOwMubt3Faum9CgUB+L/Y0DQEw1FRFUVHNhOtNex+01gF3A84GGsYsuxUoGmfdpcAcwAiciTPgutb12A+AP49ZfyPw4Ilef+HChTrYbdmyxd9NmLKniip1zr2bdEt3v9Z6ZvswOGTX87/3uv7RPw56qHXTF4zvxVj+2oc264DOuXeT/n+bD814W+PtA7BLT+MY5c1jxLNbD+ucezfpn7xxUO+qbpvRtnZVt+n8/35d59y7SS+87x+jtiefy8Ax2f0480f/1l/94y6PvvZjb5bpvP9+XfcODM1oO6HwXnj6GPHI6x/pBd/7hx4YtPuk/Z7mjff0LzuP6px7N+ndR9un/Nwn3q7QOfdu0n22yX9WQ/VzqbXzPGfx99/QD7x6YMrb7Oyz6Zx7N+knt1TOsHWTF8jvxc//VaFzv7tJW/sHT7jedI8RAZlGCFiB+DHL4oFPjOLVWn+kta7XWtu11h8ATwBXT3U7wv/2HO0gO9lCSmzkjLdlMhrITYmRAaBBbrerOmU4FcfYd6yT2QlR3POZmU/KXVzVisPVsW8L8kmew92x9l7qOvo4I9czKYRuK7ISsTs0pfWS7uZpK7ISsdkdlDV0+bspAeO1vfXkpFhYOck5oUZyV+A8KqlvgLN0e9+gfVrVm+OjIsiIj+JQk5wOg7M4Rn5aLDFeKI4BATpmC6gATEqpBSOWrQRKJ/FcjTMDB9f6K9Tosk0rJrkd4WN7amc2mfFY89NjpRJUkPuwpgOjQbFiGj/MwcqTg3QL81Iwm5yHeYNSQT3Jc7hzFzmZyXQA43Gf9Mp8W563Qv62ozR3D/B+ZQuXrZgzrWqaw3Ntybgt4OPiK1OtROg2X6o2D9tf1+G14hgQoMGW1roHeAl4SCkVo5RaC1wB/O/YdZVSVyilkpTTGcAdOCsQAhQBduAOpVSkUup21/LNXt8JMSUNnc7BiZ4OtmpaexgYsntsm8K3SmraWTo7Hos5UIeXelZX/yBVLT0eO+i7J3lekB5LUkxEWPUQhpodR9pItESwaNbUBsKfTHp8FBnxUVI1zwuykqJJjjGzT/62APxj/3EcGi4/dc60np8rc22NsvdYB/FRJnKnOeeeO9jSYT4PY1N3P41dA+EXbLl8DYgGmoA/AV/VWpcqpc5WSo0Mxb8EVOJMDfwD8BOt9e8BtNY24ErgBqADuBm40rVcBJDhyYw9WN57fnosDg3VLXIVLBgN2R3sPdYRNiXfAUrrnOlGp3iwJ68gJ4kvr8mmxWrjWLt8F4LV9iOtnJ6bPONpMcazIiuBfVL+3eOUUq6/rfRsAfx9bz2LM+JYOM0LBomWCOKiTJJG6LKntpOVcxOnPefeglmx9Nrs1Hf2e7hlweVAnfP76cnf3bECNtjSWrdpra/UWsdorbO11s+7lr+rtY4dsd61WusU7Zxfa7F2lncfuZ3dWusCrXW01nqV1nq3r/dFnNzu2g4ijIqls8cOsZu+fFf5d+kmD05lDd302uysCjZUzCMAACAASURBVKPemP11zhNeT19hWzPPmXrmLh0ugsvbHzVS3dpLZmKUV7a/cm4i1a29dPbKvISetiIrkUNN3fQMDPm7KX51rL2Xkpp2Lls5vV4tcAavOSkWSSMEem3OqXJmkg3kniLnUJhPar7vWCdK4dHzz7ECNtgS4WXP0Q6Wzo4nKsLosW26gy0ZtxWcdh91T2YcTsFWF5mJ0R4pEjPS4ow4EqIjZHLjIFRS085XN5YA8PyO2hlNcj0R99iiH/7jI69sP5ytzErAoT++eh6uni6qApxz/81ETnKMpBECpfVd2B16eGLy6Vjg6mEM9wvS7x1qIdlipqzBe0GnBFvC7+wOzZ5aZyEET/7QR5uNZCZGh/2BJFiV1LSTHhdJVlL0jLazcSPk5sL5559Lbq7zfqDaf6yD5Zmev7pmMChOz02WyY2DUHFVK0N255gKu907FSW1c1pCXtx1jA2/LpaAy4NWuE6GwzmVsKSmnY07nHMXffuFPTP6fGWnWDjW3seQ3eGp5gUl9xjLFXOnnwWRHGMmJcYc1udIJTXt7Kppp7XH5tVjnwRbwu9e2V3HwJCD3Uc7PP5hl2o7wevDox2syk6adj46OAOr226DmhrQWlFT47wfiAFXZ98g1a29wydnnlaYl0x1ay+NXeGdnx9sCvNScH8FIkwGr1SU3O8q+66BQZkiwKPS4iLJTIxmbxiPidt2uAV3DYaZfr5yki0MOTTHw3yc0d5jncxJiCI9bmapxfnpsRwK43OkrRVNw//35rFPgi3hd6/trQe880M/Pz2WqhYrDkd4V9sJNs3dAxxt651x9bz77oPeMen9vb3O5YGm1JVmtNxLFZHc47bkRDq4FOQkMSs+ioWzYtl4S6FXKkoW5qVg8HJAF87CvUjGPFfqoGLmn6/sFCn/Ds6eremWfB9pQXoshxq7w7YiYUaCM3PGoLx77JNgS/hdW48NBRi98GGfnx5L/6CDuo4+j21TeN9fS2oBiImc2Ri+o0enttyf9rkrInkp2Fo6J564SBPbpUhGUBkYstPY1c9nlmV4rXR/QU4SG9bkAPDUdQUyRYCHrchK5GhbL+094VkI2WxyHsc3rMme8QWDHHf597bwHbe1pbyJo229pHpgbO+C9Fi6+odotg54oGXBx+S6ynTzWfO8djELJNgSftZrG6KsoZvLVs7mzvWLPP5hn5/uqkgoRTKCRklNOz97qwKAh16b2YD97OypLfen/XWdw/PyeIPRoFidm8R26dkKKkdaenBomO/h+bXGunh5BgDGGaTtivG5J47eF6ZFMg41OQsP3Hvx4hn/vmfER2E2Gjgapj1bJTXtfOUPuwD4y86ZF8wZLpLRGJ7nSJVNVswmA9/9zMw/myciwZbwq22HW7HZHXxhdTZfP2++xz/s7tKmh8M4JznYFFe1YnelfQ7OsCDAww+DZcx8jxaLc3mg2X+s06uTKgKsyUvhcHMPzd3heRUzGLnHnLqPZd6ybI7zs7c/TAMCb1ruCrae3VoVlsVHKhutzE6IIi4qYsbbMhoUWcnRYZtGWFzVyqC7YI5j5sMu3Bekw3XcVmWTlbzUGExG74ZDEmwJv3qnopnoCCOnz/POFYUkqbYTdEamkc40rXTDBnjmGcjJAaU0OTnO+xs2eKKlntPZO8jRtl6vTqoIsGZeMiDzbQWTyiYrSkFe2sxKZp9MgiWC7GRL2Jco94ZDjVYU8F5lS1hWe6xstg6f1HtCbkoMNWE6sbFzfKWz99kTwy7S4yKxmI28vPtY2H0uwfnZzPfgZ3MiEmwJv3qnopkz81OINHlufq2x8tOkImEwcc/Dsm5hmkfSSjdsgOpq2Lz5HaqrAy/Qgo97E7zds7U8MwGL2Sgl4IPIoSYrc5MsHp2DcCKnZCZIz5YXFFe14i4/EG7VHh0OTWWTlQXpnkuDzU62cLS1JyyLOhTkJDEvLYbsZItHfh8/PNpB36CdPbWdYXchoH/QztG2XhZIsCVCWXVLDzWtvZy7KM2rr5OfHktlszUsD8zBqMI1m/2Na3PDZqC+r4KtCKOBgpwkmdw4iBxusvrkZACcwfix9r6wLeTgLYV5KcMD8SOM4VXtsb6zj16b3aM9WzkpFnpsdh57qzysggMArTUNnf2ctyjNI7+PxVWtHivLH2yqmnvQGo9+NiciwZbwm3cqmgE4d6F3g6356bF09A7SKicQQcEdbC30ckGAQLK/roO5ydEkWrxTHGOkwrwUyhu76bbJxYdAN2R3UNXS45OTAfg42D9QL71bnlSQk8QdFywA4OHPnRI2F5Hg47FAC2Z57jM86JrQ+Kmiw2HXG9PQ1Y91YMhjx4RwvhDgLtziyV7XiUiwJfzmnYpmclMsw6VcvcV9UJIiGcGhotFKXKSJ2Qkzm6wxmOyv62RFpncmMx7LPW6rot3uk9cT01fb3odtyOGTMQUAyzPjASmS4Q2fXjILgMiI8Drtcle582SBF3eBH4cOv96YQ+6/p4cChIKcJL7puhDww88tD6sLAYebrBgU5KZaTr7yDIXXt14EjP5BOx8cbvF6rxZI+fdgU9HYzYJZsagwKUHd3mOjtq3Pa5MZj7UiK5GoCANlbRJsBbrhSoQ+CrYSLWbmJkdLkQwvcE/sW9UcXvNDHWrqJjU2kiQPTmlx0TLnNAWemCQ52HjjmHD+knQAn4wLDSSVzVZyUmK8WjPATYIt4Rc7q9voH3R4fbwWwOz4KCJNBl76MDyr7QQTrTUVjd0syginFELnie0KL1cidDObDCxIj2XHcbt8HwKcr4MtcKYSHqjr8tnrhYtos5HMxGiqwuyi3yEvjDlcnZvMkow45iRGeXUi2kBU2WwlITqC1FjPBa+5ruyi6pYwuxDQ6NkqmSciwZbwi3fKmzH7KD94d20HNruDkpqOsMvvDjYtVhvtvYM+yaEOFO5ga/kc3wRbJTXtHDzeTadN8+Vn5fsQyCqbrMyKjyTeA/MTTdbyzASOtvXS2Tvos9cMF3lpMRwOo54trTWVjVaPjtdyW5OXQnvvIKfO9U36daCobHQGr57M/IiJNDErPpIjLeFTTn/I7qC61XfjYSXYEn7xTkUza/KSsZhNXn+tcK62E2zCsTjG1opmEi0RPktzLa5qxeH6Qtjk+xDQKpu6fdqrBVIkw5vyUmOoCqPKuI1dA3QPDHmlmubSOfH02uxUt4ZP8Aqen7PMLTclhiMt4dPrWtPWy6Bde32yeDcJtoTP1XX0cajJ6pPxWuCsthNhdF4FMoVZtZ1gMxxsZfj2BNNfSmra2XGkjY7eQZ/1uhbmpWA2OQ/9Gt+lL4qp0VpzuLnHZycDbu4eVimS4Xl5abH02Ow0uQo8hDp3tTdPFXMYadkcZzGX0vrwSXlttQ7Q1mPzSrCVlxZDdWv49Gy5C414o9d1PAEbbCmlkpVSLyulepRSNUqpL0+w3t1KqQNKqW6l1BGl1N1jHq9WSvUppayu21u+2QMxka0+KvnuVpCTxKNXrwTg1nPywiq/O9hUNHaTaIkgLTbS303xiXcqmnw+2WlBThIbbynkgmxnr/LB4+FzshJMhks8+7iXNynGTGZitARbXpCX5hwbczhMxm1584R2QXocEUZFaRj1wHpzDGduSgxtPbawSR92fwfzpWeLXwI2YBawAXhKKbVsnPUUcAOQBHwGuF0p9aUx61ymtY513dZ7s9Hi5F7ZXUdcpImuPt99qS9fOYf4KBOt1vC4ohisKhqtLJwVFzaVCGfFOcvbG5Rvq2oV5CRx/dJIzsxP4bn3qrENOXzyumLyDnmhZPZkOYtkhM9JrK/kud7LcKlIeKjJSpIlghQPViJ0M5sMLJwVx0dh1LPlTjVf4IULMLmuaplHwiQt81BjN3MSooiJ9P5QFgjQYEspFQNcBfxAa23VWr8H/B24fuy6Wuufaq0/1FoPaa3LgVeBtb5tcfjZWd3GL7ccmnLa0weVLWw/0kb3wBAbfrPdZ4PzDQbFqpwkKQYQwNyVCBf6qFs/ELh7tW49O88vVbVuPSePhq5+Xt9f79PXFSfnj0qEbqdkJVDT2kunDy+IhYPZ8VFERRjCJtiqbOpmQbr3Lp4tmxNPaX1X2IyBO9RoxWI2MscLc1DmpYZXRcLKZqtPswZ8E9JN3UJgSGtdMWLZXuDcEz1JOb/RZwO/GvPQRqWUAdgN3K213jvOc28DbgNIS0ujqKho+q0PAFar1Wv7UNoyxGO7BtCA2VDBPadHMT/p5PMU1Fkd/GRH3/B926CDP729k+788a96eXofkh02ihoHef1fW4iJ8F3PiTffC1/xxT609Tvo7h9CdTVQVOT5dLqZ7oM3jhGbPxog2gSF0Q10H2mk6MiMNzlpVquVmPpSMmMVj/9jP4kdh4KuRzEUvlsw/n5sLR0gJgIO7PrA5++Lo2UIgI3/2MrSlMnNQRMK74UvjhHpUbCr/ChFcU3Tfh1fmcnfQ2vNR3W9nJ5h8trnwtwzSFuPjZff3EJy1Ph9B6H0udxZ3sesKHjnnXc8/ho2u0YBW3aVkth5yOPbh8B5LxxaU9HQy7qsqX82p7sPgRpsxQJj+4Y7gZOFoQ/i7K377YhlG4APcaYbfhN4Uym1WGvdMfKJWutngGcAFi1apNetWzfdtgeEoqIiprMPJTXtFFe1UpiXMu5Vdq01P3niXTTOdLwhDQOJOaxbN3/CbWqt+euuY/x//z6A2WjCbHRgdziIMBm49tOnT3g1f7r7MBHz3BZeOrQdy9xlrFuc7rHtnoyn98MffLEP71Q0Q9EOLjlrlVfS6Wa6D944RjxZto2lmZrzzjtzxtuaKvff41txtdz94j5MWcs5e4FvxlF6Sih8t2D8/XiybBtL5vjns7Gix8bPSv6FKS2XdefkT+o5ofBe+OIYsaL+Q/Yd6wyKv9VM/h4t1gF63nybc09dyLq18zzbMJfY6jb+eHAb8dnLWLd01rjrhNLn8rsf/Jsz81NYt+5Ur7zOnJ2bIS6JdetO88r2A+W9qG3rxfbmFtatWsK6NdlTeu509yEg0wgBKxA/Zlk80D3RE5RSt+Mcu3WJ1np4YI7W+n2tdZ/Wuldr/SOgA2fvlxijpKadL/5qG4++WT7h/DsvlhzjYEM3RsPHV1pPdGL8XmULn/3Fu9zzt32syk7i7TvP5U+3FXLn+kU+T5s6dW4iRoOSVMIAVdEQXmXftdaUNXT5fQLny0+dQ3pcJM9srfJrO8Ro3irxPBnJw0Uywmc8jK/kpcVyrL2XgSG7v5viVcPFMbw4Z+KS2fEoFR4VCbv7B2no6me+F9Ps89JiOBIGaYQfj33z3fE1UIOtCsCklFowYtlKoHS8lZVSNwPfBS7QWh87ybY1zl4uMcZ7h5oZcjhznweGHPz7YOOoxw83W7n/1VIK85L5822FnLMgDYeGQfv4g+s3lzVy/a+3c/B4NyaD4tsXLiQ9PoqCnCS+ft58n49PsZhNLJsTz66aNp++rpicisZuUmMjSfbCYOpA1NDVT1f/EIv9HGxFmozcuDaXdw+18MCrpXIxIgC09di8VuJ5spZnxkuRDC/IT4vBoaEmxMtsV7rKvnvzhDYm0sS8lJiwqEg4PIbTiwVznHNt9YT8GLhKPxQfCshgS2vdA7wEPKSUilFKrQWuAP537LpKqQ3AI8CFWuuqMY9lK6XWKqXMSqkoV1n4VOB97+9F8HHPReWORF/ZXUddh3OM1cCQnW88v5uoCAM//+JpnJ6bzDM3FJARH8WP3yj7xJdzYMjOD145MFwAQGvNjiP+D3JWZSexp7ZjwgBR+E9FkzWsimOUuXryFgVAT97y2c65lX6/rdpn832JiblPrPL9GGydkpnAkZYeuvqlSIYn5aW6KxKGdvn3Q01W4qJMpMd5dxqPpa4iGaHOFwVzclNj6O4foq3H5rXXCASVTVZSY80k+fDCbkAGWy5fA6KBJuBPwFe11qVKqbOVUiOPUj8EUoCdI+bSetr1WBzwFNAO1OEsDX+x1tr7k9kEodLj3cRHmbhz/UIevnI53f1DXPPUB2zaV8+Xny3mo+NdPHr1SjJclXCiIox8+8IF7Knt4M3Sj3vBtNZ8/+UD1HX0E2FUGH1c1vpEVucm0T/okLmFAozDoTnU2B02KYQA5a5ga3HG2Ixp39tf3zl8kcVX832Jibkng13g154tZwD+8OsHJfj2oHnDc22FdrrWoUYrC9JjvV7cZdmcBOo6+ujoDfEAodmK2WggO9nitdeYl+rcdqinElY2W302v5ZboBbIQGvdBlw5zvJ3cRbQcN+fcOSl1roUWOGVBoaYXtsQmw82cVVBJt8435m9uXJuItc+W8ztz+8GwGhQn7gScNWqLJ7ZWsVP3yzj00vSMRkN/Oa9I/y15Bh3nD+fcxeln7Dghq+527Crup0VWYl+bo1wq+voo9dmD7tgKyM+igRLhL+bQmFeCmaTgYEhB0qpgLgwEs4qm6xERxiZkxDttzY4XNkKL+ys5dU9dX6ZmiAUxUaamBUfGfITGx9qsnL+Yu8X3Fk2x3mx6qP6Ls6cn+r11/OXykYr81JjMBm910cyz9XreqSlh9W5yV57HX/S2nlh97KVc3z6uoHcsyV86N8Hm+gbtHPpio8/gMszE7hm9dyPV9L6E1e8TUYD93xmMVXNPfy15Bhbypt45B8H+cyyDL716YV+G581kdkJ0WQmRsuV2gBT0egujhFeaYT+Lo7hVpCTxPO3FpIeF0luasyUv68lNe38ckulfK88pLLJSn56DAaD/4YXHzzu/E5qpLfT0/JSY0N6rq32Hhst1gGvFsdwcwdboZ5K6JwXyru/j1lJ0RgNiuoQnti42TpAV/+Qz7MGJNgSAGzaV096XCSnj7macckps4kyGU6YCrh+6SxWZSfy4zfK+MofSpibbOHxL67064nCiRTkJLGrpi3kB4EGkwp35aow6dkatDs43GT1e3GMkQpykrjtnDwqm6zD4wMmo6SmnS8/W8zP3iqX8V4eUtlk9eng7fEU5qVgch3DTcbASAMPFXlpMVQ1W0P2N8hd7c3bwQFASmwkGfFRIV0kw2bXHG3r9foxIcJoYG5SdEinEQ4Xx/DBhYCRJNgSdPcPsqW8mc+eMntUSXdwnoBtvPXEpdqVUnz+tCw6+wax2R00dPYPXxUNRKtzk2jsGhgu/iH871CjK6Uu2v8pdb5Q3dKDze4ImJ4tt8tPnYPRoHjpw5MVdf3YB4dbGBhyOCuTSg/IjL1X2cLxzn4skf7N8i/ISeKJLznn8/nC6rkBk50QCvLSYunqH6I1RAsRfFz23TcXDJaFeJGMhh4HWvumVPm81BiOtIRupczhCwHSsyV87V8fNWIbckyYwzqZVMDO/sHhAfZD9sA+4VqV7dwPuQIfOMobu1kYYIGHNw1XIgywfU6Pi+KcBam8vLsOh2NyV92NI67PGA3SAzITJTXt3Py7nQD8dVet349Rl6yYw6lzE9klx0qPynMVyQjVVML3K5uJMCoaOvt98nrL5sRzuNlKny005y6rtzqPxb4IEHJTY6gO4fLv2w63YjYaqGv3bUApwZZg077jZCZGsyp7+gUjCvNSiIw4cbphoFicEUeM2ej3ExnhZHdoKpusLPRj5TVfK3dNDO7PeZQm8vlVWRzv7J/0BZOi8hZSY83EmI0szoiTHpAZKK5qZXDIOS2F3fHJMbL+cMWpczh4vGt4XKWYufwQLv9eUtPOGwcaGLRrrvvNdp/8zi6dk4BDQ1lDaPZu1fc4MChnr5O3zUuNoW/QTmPXgNdfy9dKatp5s7QBm93BBh99Nt0k2Apznb2DvHuomUtWzJ5RidaCnCQ23nLidMNAYTIaOC07iV3VEmwFgqNtvQwMOcKuZ2teagyRJqO/m/IJFy6dRVykib99WHfSdQ/UdbKjuo3/OjefOy5YwL66TvbUdviglaGpMC9leKJDc4BctLp0hTO19NU9J/88iMnJTIrGbDJQFYJjY4qrWnF3ivsqrTjUi2TUWx3kpPjm98Id0IXiuK3iqhaffzbdPB5sKaV+oZTaNM7yeKXUg0qpJSOWfUsptV8pJUGfn7xZ6rwCdemK2TPeVqBVHjyRVTlJlDV0YR0Y8ndTwt4bB44DH5eaDgfljV0Bl0LoFhVh5JIVs3njwHF6bSf+fjz3/hEsZiPXrJ7LhsIc4qNMPLml0kctDT0J0Sa0hvMWpQXMRau0uEjWzk/l1T31IZta5GtGgyI3xRKSPVtLXYGPwndZLllJ0SRER4RusNXj8Nm8ULkpzmArFCsSzktx/g19+dl082iQo5TKB/4LeHCch1cDDwAjR8D/CkgD/sOT7RCT99q+erKTLZzimsAyXKzOScKhYc9RuQrvTyU17Tz+VgUAD75aGhapndaBIWrb+lgcwJUXP78qi16bnTdLGyZcp6m7n9f21nNNQRYJ0RHERpq4ce083vqoUVLOpunl3XUYFPzkqhUBEWi5XbFyDsfa+8Li++kroVr+PcrV+3J1QZbPLhgopVg6O56PQrAi4Y4jrRy3auKifFMwZ05iNGajISR7ttyXiq4vzPH5xSxP9yh9C9irtd41zmOnAQPAR+4FWus+4A/AXR5uh5iELWWNvFfZwuqcJK/P8h5oTnONT3uySOYG8qfiqlaGXP36gwFeWMVT3IFIoPZsgfNixNzkaF46QSrhxuKjDNo1N679eF75m87MxWI28nTRYV80M6Q4HJpXdtdz1oI00uOj/N2cUS5ankFUhIFX99T7uykhIz89hqNtvQzaHf5uikeVu8ZN3X3RIp+ezKbFmTlQ38WOI6HzG1JS0871v9mBxjk9jy/OVYwGRU6KJSSDrX11HUQYFfddusTnF7MmFWwppeYrpQaVUg+NWf6UUqpbKbVaKRUJXAc8P87zDwKPAZHAoFJKK6X+5nr4z8BSpdSZM9oTMSUlNe3c9r8laA2b9h8Pu4CjotGKAj443CpzA/lR4byP53UL9MIqnlLuqkS4OCPezy2ZmMGg+NxpWbxX2TJuRbH+QTsbt9dw/uL0UYO2k2LMfPmMbF7dW09tW+iWD/aGXTXt1HX08bnTxq8K60+xkSY+vWQWr+8/HnLBgb/kpcYy5HDOnxRKyhutJFoiSIuL9Nlruoty2B2a63+zI2R+z4urWrH5oWCOuyJhqDlQ18nijHi/jJWeVLClta4Efg18SymVAqCUuh+4GficqyerEEgE3h1nEzcAVcBrwKdctztdj+0BuoHPTH83xFQVV7UyaHf2KNjDpEdhpJH7OzAYfvsfKNLinFfw1y+dFTBjVLytvKEbi9lIVlK0v5tyQp8/LROt4d6/7fvEyctre+tpsdq4eUSvltstZ+dhVIpfbZXeral4efcxLGYjFy3L8HdTxnXFqZm09dh491Czv5sSEkK1/Ht5QxeLZsX5NFumuKoVewhmSBTmpaD8UDBnXmoMNa29w3/TUKC1Zv+xTpb7acjMVNIIHwKMwHeVUrfgHH91vdb6bdfjhThTIveN89y9QBawWWtd7LrVAGitHa7HC6e5D2IaCvNShufFCpcehZHcperB+aGdFe+7q3DiYyVH2wD49oULwyLQAmd54oWz4jAYAjt1t7XHhlLwTkUzX/zVtuHxW1prfvt+NQtnxbJ2/iePGxkJUVxVkMWfd9by03+WhcxVZm+y2TWb9h3nomUZWMz+ncx4IucuTCPREiGphB6S5yp48MfimpD5jmitqWi0stjHKdKFeSlEGJ2/5walQuZ85pTMBKJMRnLjDT69GDkvNQab3UF9R59PXs8Xjrb10tU/5Lf6BJMOtrTWx4GfA98Angbu0Fq/MGKVOUCX1nq8KdGXAWbgwwk23+x6vvCReakxaJw/oOHSozCSu1T97efNJ9ESwa/fPTLcXS98p6SmndhIEwsDuFiEJ2mtKW/o9vnJyHQUV7V+PFG5Q/Nf/1vCrX/Yxf2vlvLR8S4uWJw+4dXrs+anMGTXPFl0WNJ0J2Fvs53u/iGuPC3T302ZkNlk4LOnzOaN/Q38/F8V8p7OUGWTsxLhOxXNIfMdqevowzow5PNpPApyknj+ljWYTQYuXDorZM5ndlW30Tto5/L8CJ/uUyhWJNxf5yyesiIrwIMtl0M4x11t01r/csxjUTgLYIxnFc4OhD0TPN4HBHZOTYhxf/C+ck5eyByYpqogJ4m7LlrEo1evpKyhm/+3+ZC/mxR2Smo6OC07EWOA9/J4SrN1gPbewYAujuFWmJeC2eScqDzSZOBzp2Wy40gr/1tcA8Bv36+e8ASxurV3OFCz+Xg+k2C0rX7IWWI9P7CvyC/JiMNmd/DEvw+FTIDgLyO/E76e88dbPh6P6vvjW0FuMvPTYukftPv8tb3l32VNmE0Glqb4doyRexzu7z+Y+BgfbPYf68RsNPjtwu6kgy2l1AU4S7VvA9YqpVaMWaUV55it8ZwGHNZaTzQJQjLQMtm2iJk74Aq2loVZyffxXLh0Fp9flckviw6z75iUgvcV68AQ5Q1drMoOn2DffTISDMHWyInKn7+1kMe/eCr/eVbecBB1orERhXkpRJqcPy8O7Z+Tr2DR0Wtjb7Ody1fOwWQM7CknO/sGAeeV01AJEPylMC8Fo6tnOFRS+ctcxzd/ndDOTY6mtj10Ut82lzXxqbwUoky+vRhZ1+4s2vLvg00hc1Flf10ni2fHYTb55xg72WqEq4CXcRbJWAccBX40ZrUywKyUyhpnE0sZUfJ9HPOA8sm0RXjG/mOd5KRYSIiOOPnKYeCBy5aRGmvmaxs/5Bf/lhQZX9hb24FDOyeYDhfBUIlwpLETla+dn0pkhLO360QniAU5SWy8tZCb1+YSYVT8YVsNjhAabO1Jm/Ydx67hcwGcQuj2qfzU4V7oCGNoBAj+UpCTxIbCbAB+dV1BSGSYVDR2k5kYTVyUf84r5iZZqG3rDYnJt6uarRxp6eGCJek+f+3iI86x1KFyUUVrzf46/xXHgEkE2BgSlgAAIABJREFUW0qp+cAbwFvAN1xjsv4H+KxS6pwRq251/XvGOJvpAFYqpS5SShW6Kxq6tp8ILBzxfOED/v7gBZqE6AhuPTuPY+19PP6vQ3zpmW1sKW8CnOOKfrlF5uPytJKadpSCU+dO1CEeet6vbCHGbAzaOUxG9nadbKxnQU4S91+2jAcuW8Y7Fc389oNq3zU0iPyxuIY4MwwEQfpTQU4SP7hkCQB3XDA/JAIEfzpnQRoAcSFy0bO8oduvvfbZKRYGhhw0d080oiV4bC5znn+ct8j3wVZhXgruzP5Q6HWtae2l24/FMeAkwZZSKgNnkHUQ2OCqHAjOiYjLgB+719VaVwM7gMvG2dT9QCPwCs40xCUjHrsEsOHsORM+0N5jo66jz68fvEA0MOQYkSKlufm3O7nkF1v54q+28dhb5Vz7TDG/++AIO4608fz2o/zglQO89OExKpusHG3t5e2PGvm/mw+NCspKatrZdNgmgdo4SmraWZgeFza9qyU17RSVN9Njswd1asbY3q6T2bAmmwuXzuInb5RRWt/p5dYFjpNdpBkYsvPIPw5S1tBNtw02/GZ7UHwmrivMIdESQUWj1d9NCXqhVP590O7gcLPVr8HW3CQLALXtwT932eayJhbOimVussXnr12Qk8SXznD2uj57w+qgv6jirlHgz3PeE9aY1Vo3AHnjLLczOmByewp4Qin1da1174j1DwBrJniZ64C/aq1H9VMqpZKB3wDrcY7n+m+t9XgTJiucQd8trkW/Br6rXf3ISqlTXdtZgjNo/E+t9USFOsLCgXr/f/ACkbsc/OCQA6PRwBUrZ7OlvJkhV/qTze7gwb+PzoZ1FwsY6XEqODU7kdkJUbxV2ojdodlUXRyWVR8n4nBodh9t55IV4VOE9PV99biTW9ypGeHweVBK8ZOrVnDxE1u57Q+7+MLquZy1IC2k933HkVaufXY7DocmwmjgdzedzpnzUympaWdrRTMdvTbeONBA04gr8MHymTAZDVy4ZBb/PNCAbcjhtzEQoWBusgWTQVHVHPyB65GWHgbtmkV+rCw7N9lZZ622rY+CHL81Y8a6+gfZcaSNW87+xOm3z5y/KJ3ntx8N2KkopmJ/nX+LY8DUqxGezB+BeuBrk1nZFQidjzMtcaxf4uzxmgVsAJ5SSi0bZ73bgCuBlcAKnD1rX3Ft3wy86mpXEvB74FXX8rDljvKXz5Fga6SRKVJ/urWQR685lV9dv5pIkwGDArPRwENXLOMLq7OGJxo0KLh0xWw+uzxjuFdMA3Xtfbz9URNDDh0yec+edLjZSlf/EKuywyeFsK3HWVzgZOOdQlFyjJmvnptPXUc/P3879CvZPbnlMHbXd99md3Ddb7Zz8c+38oVfbeOJfx/i99tqmBUfyQOXLiUqwoCB4PpMfGZ5Bt0DQ3xwWOpazUSE0UB2siUkerbKAqD4T5a7Z6stuHu23q1oYcih/TJey839PlY0dvutDZ6y/1gnS/xYHANO0rM1VVrrIaXUTThLvU9GBnCj1rpy5EKlVAxwFbBca20F3lNK/R24HvjumG38B/AzrfUx13N/BtyKcy6wdTj38eeunq5fKKXuwhng/XMauxgSDtR1kp1sIcESHulbU1GQkzTqynJBThLP31pIcVUrhXkpFOQkUVLTzt/31jM45CDCZOCmtfMA2FzeNLzsqesK0Fpz9dPbgOA6kfIF94l2oF/F9xS7Q1Nc1crqnCTOW5w+/FkKJz0255ikkRcfQvFv0NDZT3FVKwYFCjAaFBefMpvtR9qwu3rJDQo+s3w2N501jxVzE/nT2zu59tOnB83fY+38VGLMRt4sbWCdH8aUhJK8tBiqWoK/Z6u8oQujQQ2nRvpDVISR9LhIjgZ5sLW5rIlESwSn+XE8c2ZiNNERxuGiTsHK4dAcqO/k8pV+zqLRWgfcDWep+N4xy+4CXhtn3U5gzYj7q4Fu1/+/DbwxZv1NwHfG2c5twC5gl8Vi0TjPCULyNucrv9apV9zr93YE8808Z7GOL7xGm+csPuGy9C88pOd+66+jlskNnXLxN3XWNzb6vR3Arikcl6Z9jIjKPVXn3LtJWxat9ff++u1mnrNYZ9/1is65d5Oee+ffQvY7kXr5PXrunX/TloVnjjoemOcs1nPv/JvOvuuVkNj/1Mvv0Vm3/1GjDH5vi5dvXj1GJK67SWd/56Wg/zumff77evZ//tLv7Zi14ad61pce8Xs7pn1TBp11+x91yqXf8XtbMq5/XKd/8Yd+b8dMbqakOTrn3k06dsV6b77OSY8RgZpsHQuMnZOrExivfzrW9djI9WJdY7nGPjbhdrTWz2itV2utV2dlZfk94JzpbcuWLeMub+8ZICIxg4e+davf2zjdfQiE20DdQTq3vcBA3cETLnvynpswREZTsqPY720OpPdi5QVXclHBAr/vw1ToGRwj/uvHvyU+ykTr/nf8/l566z092W2g7iD/59oCAL7+6SWjvifBsg8nu713qJmYJefwnYtPoaf8/VHHg4G6g7x8x3nc89llvHzHeaP2P9D2YzK33/1/38IYk0hxZVPQ7sPYmz+OEU/95EGUyUxNc7ff938yf4+JbovWnMfnzy/0e5uvvexC8leumdY+BMKtpLoFY0wiz/3wTr/vw3WXnc/c5Ws8tj1/7Mfftuzk/2/vvuPbrO7Fj3+O5BXHdiyP2Em8Ysd2BhnECTEJZECAQge03LaUFEpboC2l7S0dtP1dKJfbW7puJ5SWVWiZHayyRxLIchI720nsxCuJHW95T0nn94ckYxxvS9Yj+ft+vfRK/OiRfI4lHT3fM74HIO/1v3utDqNh1GCrDRi4EU0UMNh45sBzo4A27fwLjOV5poQjlc4YVpJjTI6Ls+IAeP9EnY9LYhzW9h5K6to5f4psZtzWbeONI9V8bOlswoLNvi6OT33y/DnMnhFGSQCsURmox+bgrpeOkBobzlfWDb6wfayZHI1sfXY8IUEm3iisHvdzyLYakB4fAUCJH08lbOu2cbqx06fJMdySLdM429xJr90x8skGtPlYLWaTYp1rWwBfyk6MpL6tm4Y2/02lf/hMEyFBvk2OAcYNtoqBIKVUZr9jS4HCQc4tdN032HmFwBLXKJfbkiGeZ0roS44xxz82VfV3MyPDSI408X6xBFtu+09PrfVarx8+S2evnWuXG3/TWm9TSrFh/ky2n6yn22b8faXG4pHtpZTWtXPPJxZNiaB6emgQazPjeauwZtS9u/3tLm3guod28cs3i7j+4cBOmDKcQEj/fqLG98kx3JJiwnFoqGrq9HVRxuXfh6qYNSOMkwbIUJmZ4E6S4fuyjNfhymYWJEYSbPZtuGPIYEtr3Q48D9yrlJqulFoDXA38bZDT/wrcoZSao5SaDXwHeNx131bADnxTKRWqlLrddXyzN8tvZEcqm0mOmUZ0+JROyDipFseZKaiw0t5t83VRDGFfRRNmk2Jp0tTIRPj8vkrSYsNZPkVG8kZyyfyZdPTY2VPW6OuieMybhdX8+q1iLkiz+GQTUl/5yHmJVDZ19s2YGI57FGtXST2P7yjj5ify6bU7g7Rum4P3XJvITzWx00OICgvy6/Tv7iQK8xN934mbEuPOSOh/wdZbhdVUNHRQae00RMZW90jliVr/nAzmcGgKK1tYnOT7mVyGDLZcbgOmAbXAM8DXtNaFSqmLlVL9W6U/A/8GDgNHgFddx9Ba9+BMC38j0AR8CbjGdXxKOlzZLFMIJ9l5cWZ67ZpdJZL6HZwXXQtnRTEtJPB7/89YO9hV2sCnlifx4QH2qWt1RhyhQSY2H/f/i2ub3cHTu0/xtScLsDk0B880+/wCaTJtXDATs0nxRuHZYc8rqLCy6ZE8fvVmEZ97eDf3/PsoSZZphJgVJtfH4tXDZ2nu7J2EUhuLUor0+Ai/Htk6Xt1KeIiZJMs0XxelbxNgf8xI+Py+SsCZccEI28UkRIUSFRbktxkJyxvaae22GeKa17C7lWmtG3EGSgOPb8OZ+ML9swa+77oN9jz7gRwvFdOvNHf0cqqxg+suSPZ1UaaUTIuJ8BAz7xXXsXFhgq+L41M2u4MDp5v47Mqp8R58cb/zy/OT58sUQrdpIWYuzIhly/FafvzxwbZONLZ3jtbwXP5pWjp7OHq2ldauD0asbfbATWk/mOjwEC5Mj+XF/ZXUxNuJnGsdtO55pQ109zpwTza8bmUyP7t2CQUVVvJKGwgyKX71VhE3Prqbv355FTOmTa1tSdLjp7PjpP/uWVZc00pmQiQmk+87lBKjwgg2K05b/S/Yau0y1l6MSimyEyP9dq+tlw9WARBk8v24ku9LICbNkSrnei0jRPlTSbBJcWF6rCTJAF7YX0lnrx3L9MCfxqq15vl9layaG9PX2yqcLpk/k/KGDr+bOvWvgjPc/Nd83j5aw54yK7lzY/jOZVmEBpkMc4E02RbMjqKyqYt/negdcupTetz0vkArLNjEp1c4O1vcCUO+si6DBzflcPRsC9c+uJNfv100pUYIM+IjqGnpps1Pp5oXVbcy3wDJMcC5r92c6Gl+t7Gxw6E5eraF9Vnx3HF5Nk/dnGuITpvMhEiKqlvHtS7TlwoqrPxhs3ML3//34mGftycSbE0hfckxZkuwNdnWZsVT0dBBRYP/ThWZqIIKKz98/jAAf5wCGcie3Xua0vp2VhjgC9No3Oua/GkqYVl9O3e9dKTvZ5OCZSkWvnFpJk/fkmuoC6TJFOwazRhu6tML+ysJMZv46rr0If9GGxcmcMdlWZysbeP37540xJqVyZIe50ySUeaHUwnrWrtpaO8hywDJMdySY8L9LtgqrGrB2tHLJ5fPMVTG0uyESFq6bNS2+ldGwrzShr5N5I0wJVOCrSnkcGUzSZZpU2JUwWjWZjnTuE7lrIR5pQ3YXI2fe7pVoCqosPJfLzovzB/dXjZlLhpHKzkmnKyECLb4SVKE6uYuPv/IbswmNegoViCldB+rSxckEGx2BlxKqXNG9t4rruOtozX852WZ/ODKBcP+jRwa3BPRegxwgTRZ3OnfS/0w/bt7itl8AwVbSZZwTlv9K0GGe+bLmnlxPi7Jh7lTpvvbuq3cuTGAsz0xwowDCbamkCOSHMNn0mLDSY6ZxnvF/jsvf6Jy02P7LqSM0Ph5U15p/Qe9agEeWI7Xhvkz2VPW2LdOwais7T3c8Ohumjt7efrm3Ck9ijWYnFQLz956IWlRJmwOTUu/17PH5uC/Xy5kbtx0vnzR3BGfKzc9lpAg52WJaZDALVClxoajlH+mf3/nWA2AobZySIkJp7G9x6+mZW47UcfCWVHERYT6uigfkpXg7Ajwt3VbM6PCAOeIuRHaagm2pojmzl4qGjo4T4Itn1BKsTYznl0l9fTY/HOzxYmaHR2GxrlexwiNnzcluBp6o/SqGdEl2TPptWu2nzBuB8T2k/Vc+fttlDW088gXVrA4acaUHsUaSk6qhR+tCmPBrCi+/dwBKl17HD22o4zS+nbu/vhCQoNGzj6ak2rh6VtymT0jjNTY8CnzNw4LdmbyK633r2CroMLKEzvLAbjtqX2GGcFPjnFmRfSXqYTt3TYKKqxcnGWsUS2A2IhQ4iJC/G5k6+hZ53YUt63PMEQ7IsHWFPHCvjOAc3Gy8I11WfG099gN84U02fLLnfX+9sYsQzR+3lTd3AXA19ZnBHxgOV45qRaiwoIMu26roMLKjY/uprq5yxk0+3hTTKMLMSv+uGk5Nrvm60/t43RjB79/9wQbFySMae+xnFQLm3JTKalrp6aly4slNpb0uAi/Sxjz4v5KXAP4hlgX45Zsce+15R/B1u6yBnrtmrWZ8b4uyqCyEvwvI2FhVQsmZYy930CCrSmhoMLKT149BsAv3phaWZ6M5MKMWIJMaspmJSyosDIt2MyCWcaZ2+8tW4rqWJI0g+9/ZL4EWkMIMptYmxXPW0druH/zCcO1SztL6vsuJB0ObZgLSSObGzedX/zHEg6cbuKq371Pt83Bp8ax7cGlC/wvgcpEzY2bTll9u99kfXM4NLtKnKPSRsvE6c7+6i/rtradqCcs2GTY74qshEhO1LbhcPjHexPgaFUz6fERhtnPU4KtKWAqJSYwssiwYDITIvhXwRnDXVi67S1v5HfvFLOrpJ6uXjtdvXZ2lzbwSknPhMucX9HIsuRoggJ8hMDa3sP+U1bWj6E3f6pKj5tOc2cvv3672HDZ5xIjnVNBTQa7kDS6qxbP4qrFibR223E4NHf848CYX9fshEjmRE/jXdd6oKkgI346HT12qv1kNO+5/NOcrGvnW5fOM9waRkt4MBGhQX4zsrXtRD0XzI0lLNgYgcFA2YmRdPTY+6YH+4OjVS0snGWMUS0w8KbGwnPciQk0ctHgSwUVVk7UtGFzaK5/OI+nbzHOlxPA+0V13PiXPYPep4BXyvPG/YXa3m3j2NlWblufMcFSGt/7J+pwaNiQbcwpIUZid/XiO/QH05CM8plo63Eurr91bTqXLUw0TLn8wfzEKF4/XP2hdPBj+fsppdi4YCbP5Z+mq9du2ItQT+rLSFjXzqwZ03xcGuf31SslPYNuVG1t7+HnbxzngrQY/nNjFkr5fjPj/pRSJFlce20Z53p7UFVNnZysbeO6lcm+LsqQ+mck9Ic9I63tPVQ1d7FotnFe/MDuYhYALEuOxmxWrEyzGKr3aarJK23AoY2boe7B9072/V8BazPjWJvpXLA73B46o3HgdBN2h54S770tx2uJnR7C0qRoXxfF8C6Zn2DYDJUFFVZmzwgbMV25ONeaeXGEBk9so+dLFyTQ1etgx0njJlDxpPR4515bvli3VVBh5QHX3od2h+aNI2f53EN5Q25U/cu3imjtsnHvNYsMF2i5JceEc9pq/JEtd4Kgiw26Xgsg05WRsMhP1m25k2MsNFCwJSNbU0BFQzs2u+azK1PkosGHctNjCTGb6LI5DJfWuKmjhwOnmzGpDzLofWtjFuAMEnvsetA9dEYrv9yKUrA8wN9/dofmveI6NmTPxGQy5kWIkeSkWti4MIH3iur425dXGap9Kqg4t0dfjE5OqrNjL6+0gdz02HH9HVelxzA9xMw7x2q5dEGCF0ppLIlRYYSHmCmZ5PTvBRVWrn84j26bAwWYTGDvlzC3q9fBM3sqWJ4SjVKKg6ebeGbPKb60Zq5hkg8MJiUmnO0n6tHaWKnUB3r/RB0zI0P7UqwbUVRYMLNnhHHCT4KtwqpmAJlGKCaXO4tMdkLgJyYwspxUC0/dksttTxWQEBlmqAu5R7eX0dlr59efWcrZ5q4PXSA9c0suX3k8D7spaNyNV35FI9kJkUSFBXuy2IZz8EwT1o5e1s+X9Vqj9dHFs3j7aA3TQ4zzdVTZ1MnZ5i5WGOgz6m9yUi0TauNCg8yszYpn8/EatD7PsCMonqKUYm7c9ElP/76rpJ5u13YkGshJjWF5SjSPbS+n1xV1/bOgklONnXxiyWzu33KCGdOC+c+NmZNazrFKtkyjs9dOS4+vSzI0h0Oz42S9c4Tf4O/vrMRIimr8I1vm0aoWEqPCiDXQnmUyjXAKKKpuQymYN9O4PSdTRU6qhUsXJFDW0G6YzD5NHT38ZUc5H108i08tTzpnD6GctBhuWRKKtaOXv+4qH/Pz2x2a/aeaDBVcesvW47WYFKwz8JQQo1mR5nxf7C1v9HFJPpDvKsuKtBgfl2Rqu3RBAjUt3RypbPF1USZFevzkp3/v7HVuRqxwbg1z50fm84MrF/DMrblcmxnMs7fmcu/ViyiqbuW/XjpCdUs37d02ig1+4e1eW1TXadx9LQurWrB29HJxpvH21xooOyGSkto2bHbj/j3dCqtaDLVeCyTYmhKKa1pJjQk3TArMqW5p0gxau2yUNxhjA8uHt5XS3mPjm5cO3VOZHWNmXVY8D75XQktX75iev6i6lbZuW99FdSDbXFRLTqqFGeGBPYLnSUmWcGbPCDNUsLWvwkp4iJn5iTIbwJc2ZMejFLwzRbISpsdNp7Kpky5XAORtzZ29PLvnNFkJEXzn8qwPrenOSbXwsYwQVqXHcuOFaXxxTVrf+kp/2AohxR1sdRijU3Mw7m1g1swzfrCVmRBJj91BeYOx18F19dopqWsz1HotkGBrSiiqae3LJiN8b2myM3HCwTNNPi6JM2vP4zvKuWrxLLJHuLD87uXZNHX08ui2sjH9joIK1yhBamCPEtS2dHGkskVSvo/DirQY9pY3GmaPofwK65TYpsDoYiNCWZ5i4d3jUyTYip+O1kxaR9yv3yrC2tHDrz+zjNsvyRx29sHFmfETTnoymZIsxh/Zeu1wFTMjQznlBynq3ctQ/vCu8fZE7O94dSsOjYxsicnVbbNTVt8+4oW0mDzz4iOYFmzm4OlmXxeFh7eV0tFr51vDjGq5LU6awVWLE3lkWymN7aOfCJ9fYWVmZChJFt+nM/amrcXOXsoNEmyN2cq5MdS0dHPGAJuQtnXbOHa2RdZrGcSlC2ZypLKF6mb/2H9qIjL6pX/3tsKqZv6WV8Hnc1M5b86MEc93Jz0x2p5aQ5kWYiYuIpT6TmN04Az0XlEthVWt1LV2G26PwcG0uma0vHywytDlPVrlykQ4a+T39GSSYCvAlda1Y3doMmVkyzCCzCYWz5nBIR+PbG0pquXhbaWsTo8d9cjnHZdl0dlr556Xj/SlCh5JfrmVFWkWwy8AnqitRbUkRoWxYJZ81sZqpWuK6Z4y308lPHi6CYd2rlUUvrfRlYnw7peOGPYCz1PmxjnTvz+z+5RX6+pwaO5+qRBLeAjfuSx71I/LSbWcs6bXyFJiplHXYcyRrd9vdm61MtFtVSbL/tPO6xWjl7ewqpnI0CCSY4zVuSvBVoCTTITGtCRpBoVVLX3ZniZbQYWVW57Ip9euya+wjvqLfd7MSNZmxvPywbP831tFI/ZwVTd3UdnUSU6ATyHstTvYVlzPhvnxAR9UekPWzEiiwoLIr/B9sOXepuD8FNknzQhaO3tRwFtHawzdo+4Jx6ud39fbTtZ7ra4FFVa+/vQ+Ciqs3Hnl/IBeX5ocE06dAUe2yuvbOXDaitmk/GZaZm56LGbXdiZGLu/Rsy0smB1luO9hwwVbSqkYpdQLSql2pVSFUur6Yc79nlLqiFKqVSlVppT63oD7y5VSnUqpNtftLe/XwFiKqlsJMqm+HjNhDEuTo+m2OSiq9s2+FbtK6rG5siHaxrjBsntKqkNDzwg9XPl967X8oyd0vJ7efYrWbltfBiwxNiaTYkVajCFGtqbKNgX+Iq+sEfflspF71D2hf928UVf3nlqvH6lGKWdCjkAWZFI0dGr2lBnrPfOz148TFmTmzzfk+M20zJxUC3e40v3f9dGFhiyv3aE5frbVcOu1wIDBFvAA0AMkAJuAB5VSi4Y4VwE3AhbgI8DtSqnrBpzzca11hOt2ubcKbVTFNa2kx08nJMiIL/XUtTTJt0kyZs1wDrGbxtGrdvmiREJd7yeHhunDZLnML7cyLdhsuMxAE9Xc7fwCP3Smif/+dyH3/LsQgN+/Y+zFw0a2Mi2Gkrp2Gtq6fVaGqbRNgb/ITY8l2OzspTabjduj7gm56bEEuUYPgrxQ17zSBnpce2opYLcBOje8paDCyssHq9DADY/uMUy7vLu0gTcKq/nqugw2Lkjwq2mZ169KRSloGMOa7clUVt9OZ6/dUJsZuxnqClwpNR24FrhLa92mtd4OvAzcMNj5WutfaK33aa1tWusi4CVgzeSVeGQFFdZRr23xBslEaEzJMdOwhAdzyEdJMjpcqYVvvjh9zL1qOakWnr4ll6+uSyfZMo3/fe0Y/z5YNei5+05ZWZo8g+AAy+pm7dZ85s95fOL+HfxlRznuJHq9YxwlFB9wr9vK9+FFUXHN1NmmwF/kpFr40+dzALhuZbLfXJiOR06qhXs+4exb/vZlWR6va256LO7ZVSEGngrmCXmlDdhdszdGmoExWRwOzf++dozEqDBuvjjd18UZM8v0EBbOimLHyXpfF2VQhVXO66lFs42VHAMgyNcFGCALsGmti/sdOwisG+mByjlB82LgzwPuekopZQL2A9/TWh8c4vG3ArcCxMfHs3Xr1jEX/qTVzvFGO/NjzKRHmzhQa+OPB3uwOyDYBN9fGcY8y+TsddXW1sYb72zhdGMnK2Nt46qPr7W1tflluQcaqh5J4Q52HD/D1q2T37v42qEuZoQqLpxWTWtZDVtHyOY+WB1yp8Hi5Yrf7VN845n9PPT2ITamBLEk3oxSiqP1Ng6f6Wb1bLMhXseJvp/6txEhifMAWJVoZtUsM3862IPNAWYFoU0VbN16xhNF9iqjfb56HZogEzy/7SChdcdH9RhP12HzKWfGLdvZYrY2n/TY847EaK/FeHizDmYgIVxx6OQZtm713oWeJ9uI8V5HzLRrzAoOHS9hqz497rIMpsumURoyo018NjuE1rKDw7b9/vy+DG2yE6SgR2tAGaJd3lll49CZbm5ZHMLundtG/TgjvQ7JIT28U97Lm+9uIdQ8tnVR3q7Hm0U9mBVUHS+gttg7a7bGWwejBVsRwMCt4puB0QzN3INzpO4v/Y5tAvbhHDH/FvCmUmq+1vqcuVta64eAhwCys7P1+vXrx1TwggorP397F712jaIXs0n1rYkBsGvojk5l/fp5Y3re8dq6dSvRGcvgnR1ckbuE9YsSJ+X3etLWrVsZ6+tgREPVY19PEfdvOckFqy8iPGRyP4p379nChfOi2LAhZ1TnD/daxKTXs+mR3Ryut3O43k6IWREfGUpVczca2Fvj4I65S33eIz3R91P/NiJ0VqYOCzbx/U+uIifVwrpcK3mlDeSmx/q8nqNlxM/X8uJdVNsdrF8/ugkKnq7Di8/uJz6ygf+4csOkLrA24msxVt6uw7r6g7xZWMPateswmbzz2niyjRjPdYTbkqId1GnF+vWrx12Wwbxx5Cy6qKe1AAAgAElEQVR29nHvpy/gwoyRR7X8+X25Hjh/uZUfPptHSZODz165lshJWodZUHHu90FXr50f/Wor582J4oefu2hM72FDvQ6zannjL3sJTzmPizPjx/RQb9fj0ZLdzJ/Vw8ZLLvba7xhvHSZ1bo9SaqtSSg9x2w60AQMnW0YBw2YRUErdjnPt1ke11n0T/rXWO7TWnVrrDq31fUATztEvj3txfyW9dmdwpYHlKdHcuja9b/61BnLnTm5GtuJqyURoZEuTo3FoOFI5sH/Bu2pbuzjV2OGxoGDfqQ/6LhSwJCma8JCgvql1doc2xBQOT7KEqg9Nv/S3lMhGtXKuhcLKZjp6bD75/fkVVlakBv42Bf5oZVoMzZ29nKht83VRvC4nxcLBM81966s85e2jtcyYFjxlpsnmpFr4VGYIdg3bT0zO1LfNx2v49J928ss3i/j0n3byvX8c4JVDVdz5r0NUNXfx6Zxkr3UWTIaVaTEEmRQ7ThrrO11rzdGqFkOu14JJDra01uu11mqI20VAMRCklOq/w+pSoHCo51RKfQn4AXCp1nqkMWLneLKH9dgcvFfk3NDUrCAs2MSdVy7gR1ct4LmvXMjF8+Jw6A/2KZgsRTWthAWbJEOaQS1xJcmY7P229lU4f99yDwUGuemxhASZnFPogk388KoF/OzaJYQFm/wmre1YzQhVElh5wYq0GGwOzYFTk584pqalizPWTnldDeoCV2flnvLATerglpNqocfm6FuD4gl2h2bz8Ro2ZMcH3Bra4cyLNhEZFsTm47Ve/13W9h7u/Och3JOaHBr+WVDJ7U/v56UDznXN971+zDDJOsZjemgQ56dEs7NkYsGrp/MZ1LR009DeY8hMhGCwaYRa63al1PPAvUqpm4FlwNXAoGPpSqlNwE+BDVrr0gH3pQDJwF6cQeU3gDhgh6fL/cCWk5yydvCjq+bTa9cfGjrOSbXwxJcu4GtPFXDf68dZODuK1Rlxni7CoIprWsmcGdm3N4IwlvjIUGbPCOPgmclNkrHvlJUQs4nz5nimUcpJtfDUzbnnTJsY7JgQw8lJtaCU84J69bzJaSfd/pHvXB8TEWaor0XhkhITzszIUPaWNXJDbqqvi+NV7vayoMLK+SmemoFgxdrRy8aFCR55Pn8RZFKszYpnS1EdDof22qhSc0cvn390N9bOXkLMJuwOB8FBJv5y00reOFLNX3dVfGhDYH/+TrwwI477N5+guaN3XPu0FVRY2fRIHt29DkKDTDx1y9iSdA02TfPlg5UABJmM2ZFgxG+V24DHgFqgAfia1roQQCl1MfC61jrCde5PgFhgb79pH09qrb+Kc53Xg0AG0AUcAK7UWnt07PPY2RYe2HKSa5bN5ta1GYOeYzIp/u8zy7jmgR3c/vR+/v2Ni5gT7f3drYuqW8c8p1ZMrqXJ0Ryc5BHPfRVWzpsTRWiQ55K15KRazmksBzsmxHCiwoKZnxhFfvnk9vwWVFj5zTsnALjn5UIyZ0bKe9dglFKsnBvD3vJGtNYBPdVzZlQYyTHTKKiwcrOHFj68c7SGYLMz8JhqLsmeyauHzlJY1cLipPFnqttT1sCukgYuyoz/UPvQ0tXLjY/t5kRNGw/fuIKosOAPBQMhQWaeyz9Nr80REDM91mTE8vt3T5BX1sAV48gHkFfaQHevAw102Ry8eqhq1O3tq4eq+MYz+3Fo5zS18+ZEYQkPYbsrQ+JPXj3KgtlRhmu/DRdsaa0bgWuGuG8bziQa7p/nDvM8hcASjxewH5vdwff/eYjo8GB+/PGhtgJziggN4s835HDN/Tu44ZHdXL1s9jkfWE9q69HUtnaTnRgx8snCZ5YkRfP6kWqs7T1Ypod4/fd12+wcqmzmCxcGds+w8F8XpFl4du9p/rD5BKsz4iblS3NnSX1fmuhA6HkOVBekxfDqobOcsXYG/PT4nBQLO0oaPBZYvn2shtz02Cm5Wff67HiUgs3Ha8cdbBWUN3LdQ3k4NPz23RNcvXQOVy+bjc2u+fG/j1DT3MWfbljBhuyZAB9qP4aa/eGvzk+xEBZsYlfJ+IKtZMs0dL+fn8yrIDMhkutWJp/zXnePYi2cFcl7xfX8dVd53zRNDTS09VDR0NF3zL39itH+xoYLtvzJI9vLOFzZzAPXLx/VhXJGfAS3XzKP+14/zm/fOcGD75V4befwyjbnwlrZY8vYliY7G/6DZ5pY72qkvamwqoUem8NwDZEQbrERoXTbHPzm7WIe2HLSa21kf2bXF/x4NvkWk2dlmnPd1t7yxsAPtlItvHigyiOBZUldG6V17XzhwjTPFM7PxEaEsjQpms1FtXxrY+bIDxjEM3tPfXCRr53T1l48UNl3f7BZETPMdWAgzfQICTKxMi1mXPttORyaJ/NOER5i5qbVaSxPtfD4jnJ++Pxh/n2gihVpFtZlz+S8OVFsLarjG8/sp9fmHAUzKbh0fgLvn6jDZneOEv7h+uUAbHokz9AjhxJsjdO/D1byyzeLWDXXwlWLRx/Zu9PBe3vu7hlXsJWdKMGWkS2eMwOl4NCZ5kkJtva5FqMu99A6ACE8rdO14bZDT94o03vFdcRFhHDT6jQunKTRNDF22YmRRIYFsbe8kU8tT/J1cbwqJ9UZWBZUWCccbL17rAaASxd4/zvGqC6ZP5PfvFNMfVs3cRGhY3qsw6EpqLCi+KBD5rEvrOSlA5X8Pf8M2nWOEUdUvGXNvDh+9vpxalu6mBkVNurH/T3/NHvKG/n5tYv57MoUwDnN8+6XCnlydwU7Sxv4/ebB9zi8ac1c7v7YwkHXbBl95NCYK8kMrqDCyreePYDdoTlwuvlDqa9Hkpse25cOPsjsvQi8stVBZGgQiWP4EIjJFxkWTEZ8xKSt23J+cU8bU+MoxGTauCChL2XsZPRSHq1qYXdZI7euTef2SzIN+UUtnMwmxYpUC3vKAj8jYXZiJNNDzB7J1vbO0VoWzIoiyRLYo4HDuWT+TLSGra7M0WPx2pGzlNV38I1L5nHH5dk8dXMuq+fF8ZmVKYQGcNbd4ax27dO2awzbujR1O/jpa8dYNTeGz6xI7jtuMilmRYfhzl2igIvmxfHVdekEmxUmV5bvjy6eBQy+1YrRt1+RYGscnus3nGxzzQ8drZxUC7/57FIAbshN9dob40ybg6zEyIBeRBwo5kSHsau0gQIvpzTWWpNfYSVHRrWEgeWkWrh0wUxCg0z87curvP7l+cTOcqYFm/nsihSv/h7hGSvnxlBS105DW/fIJ/sxs0lxfoplwsGWtb2H/IpGLpvCo1oAi2ZHMTMylC1jTAFvszv49VvFZCVE8K2NWR+6oHevxXIHYEa90PeGRbNnEBUWNKaphE8f66HL5uC+Ty0+59p04BYy374six9cuYBnb72Q7wTA31eCrTHSWrPfNZI13t6Mjy+dQ3rcdIpqht2redy01lS2OWS9lh8oqLCy42QDHT12rn9kt1f33zhj7aSutduvGywxNXx0ySy6bQ7CQzyXMXMwje09vHigkk8unzOuFMZi8l3Qt27Lf/cqGq2cVAvHq1to6x7/Jt9bimpxaKZcyveBlFJsyJ7J+8V19NpHv1n0PwvOUFrfzncuzx50Gx2jj6h4i9mkuDAjlh0nnUlcRvLQ+yXsqbZz7flzSI8/N3HbUIFroPx9JdgahLVLD3nR++6xWk7UtvG1dekT6s24dMFMdpc2TqgRHcq7x2pp74XwYHl5jS6vtAGHq6HqsY1tlHSs9p1yvqc9tW+LEN6yaq6zA2t3qXdHe5/Zc4pum4ObVqd59fcIz1mcNIOQIBN7p8jmxg7NhDb5/kf+aSJCzfTYRh9gBKoN82fS2m0b9dYSXb12fvfuCZYlR3P5FA9WB7M6I47Kpk7ue+34sB3F207Ucd9rxwF44UDlkOcGSmA1GLkaH0Rzj+b6h/POeUM4HJpfvVVEWmw4d1yePaE3xcYFCfTYHWwrHvv84eEUVFi57al9APwt75Rf71Q+FbiHzgFQeHXO974KK+EhZuZL0hRhcLOjp5EcM43dZd7rfOi1O3gyr4I182JlFoAfCQ0ysyw5ekoEW8tSolGKcX+P55XWs6u0kbZuO59/1LszJ/zBRZlxmE3w67eLRvW3eDKvgrPNXXz/imxZkjEIi2s2wMPbStn0yLnXzOCchnn3S4V9qd57vdypbFQSbA2h2+boy+Dj9srhsxyvbuXbl2URbJ7Yny4n1cKMacG8PeB3TFReaUPfELndMTXf1P7EPXS+LiserWFm5NiyJI1FwSkry5KjCZrge1eIybBqbiy7yxpxOEaeojIebxXWcLa5iy+uHnK7RmFQq+bGUFg1sel1/iAqLJjshEgKTo0vSHoq71Tf/6fqRW5/RdWtaO2cgjpUcOC2/WQ9//dWMUvmzGD1vLhJLKX/OG3tAJzZtYeamfPT145TVt/uTHTB1Esk4iZXXcN49fBZmjt7AWd0/pu3i5mfGMnHl8ye8HMHmU1syI5na1Fd32aanpCbHou7A2aqvqn9TU6qhZ9ccx4Arx8565Xf0d5t49jZ1oAcnheBKTc9lqaOXoprvbO29fGdZaTEhLNh/tROHOCPVqbFYHfovq0sAllOqoX9FdYxdzporSmsakEx/vXlgSavtAH38qLuXge7SgZP7pBf3sgXHttDZ6+d4zWtU35EcCi56XGEumbmODSEDOjIfXr3KR7bUcYX16Tx7K0X8qnMYL9PdDFeEmwNwhKquOtjC6hq6uTLj++ls8fOv/adocy1SNI0yCLJ8bh0QQKN7T3sH2ev1WDOT45meoiZtCjTlH1T+6PkmHAWz5nBa4ervfL8B880YXdolsv7QfiJVXOdiRC8sW7rH/mn2Vtu5ZL58YMuehfGtjzVggIe3Hoy4C+Ec1IttHbbuPeVo2Oq667SBkrr27l17cTWlweS3PRYQoNNKJyjMQdON53T2d3RY+NHLxzuO24fY8bpqSQn1cLTt+TyzUvmkR4Xzi/fLGJLkTPb486Seu5+6Qhrs+L5f1ctICfVwscyQqbse1CCrUHMCFV8+aJ0fvvZ8yk4ZeX6R/L4ySvHyJwZwUYPpk9dlx1PkEnxzrGxpSIdTkldG63ddi5JCZqyb2p/deXiRA6cbqKyqdPjz/3KwSoAzMiFpfAPyTHhzIn2/LqtggorP3j+MADP7Dkd8BfrgaiouhUU7CptHHE6mL8LC3Jm5HxiZ/mY6vrQ+6XETg/h25dlBWzSgbFyT9v/7hVZfDpnDu8cq+W7/ziIzbX0oqqpk0//aRcnatoIMikZERyFnFQLd1yezfO3rSEzIYKv/K2An752jC89vpeEGWHcf/35snQBCbaG9dEls/jK2nT2n2qitdtGRUPHmDYwHklUWDCr0mPOWRs2Ee50uFkW76ZMFp535XnODfveOOLZ0a2CCivP7j0NwK1P5gf0hYkILKvmxrC7tHFUqYVHK6+0oa/Xeqz7JApj6D8dLNDXIpU1tAHOkZjR1rWoupWtRXV8YXUaYcFyLdCfM+NdJr/89DK+d0U2L+yv5MZH9/DD5w9z5e+2caqhg8duWslzX7lQRgTHIDo8hKduXsWc6DAeer+Url4H9a3dnKhp83XRDEGCrRFEhgX3jQV4I+HEpfMTOFHbRkVDu0eeL7+8kbiIEBLCZQTD38yNm86CWVG8dtiz67bySuv7NuEO9AsTEVhWpcfQ0N7DyVrPfWEvnB0FgEJ6rf1VbnoswWbnd1yQObBfw9z0uL6prqOt60PvlzIt2MwNuaneLp5f+/qGedy0OpWdpQ08s+cULZ29/OSa89gwf2ZApyH3lujwED7WL6eBdGZ9QIKtEbjn+HprOHnjAufeDZ6aSrinvJEVqTGSptRPXXVeIgUVVqqbuzz2nCkx4YBcXAr/436v5pV5bt2Wze7sebjughTptfZTOakWfvUfSwG4ZW16QL+GOakWfnHtEgBuvDB1xLpWN3fx8sFKPrMiCcv0kMkool+Ljwzr61A3KTjjhWn8U8n67JmEefGa2V9JsDWCoXa19pSU2HAyZ0Z4ZCrh2eZOzlg7WelaWC78z1VL3FMJPTe61WNzXlzetCZNLi6FX0mJCScxKozdHuwdzS9vJMRs4scfXyifBT/2saWzmR5ipsWVMTiQXZuTxPzEyL5lAsP5y44y7A7NzRenT0LJ/J+3O9SnGm9fM/urIF8XwB/kpFq8+oa5dEECD79fwq/fLmJd1sxx/y73rugr0yw0nqzwZBHFJMmIjyA7IZLXjlRz0xrP7P+z/7SVyNAg7vroQo9l0hRiMiilWJUew46TDWitPTJiv7e8kcVJM2Qti58zmxTnzZnBoTPNvi7KpPjk+XO47/XjlNa1kR4fMeg5rV29PL37FFcunkWya0aDGJ47OMgrbSA3PVaCAw/w9jWzP5KRLQNIiZmGXcMf3j05ocxKe8sbCQ8xs3BWlIdLKCbTlYsT2VveSG2rZ6YS7j/VxNLkaAm0hF9aNTeW+rZuSusnvq61q9fO4cpmVqTJhUAgWJI0g6NnW+ixOXxdFK+7etkclIIX91cOec4v3yyitdvGukzZhHcsZH2W8DbDBVtKqRil1AtKqXalVIVS6vphzr1HKdWrlGrrd0vvd/8ypVSBUqrD9e+yyanF2DS29wBjyzY0mL3lVpanWCTNpp+7avEstIYfPX94wpkDO3psHK9u5fyUaA+VTojJtSrdc/ttHTrTTK9dszJVploHgiVJ0fTYHBTXeGfjayNJnBHGmow4XjhQOWh2zl0l9fx1l3NGy90vF0rWWSEMxIhX5Q8APUACsAl4UCm1aJjzn9NaR/S7lQIopUKAl4AnAQvwBPCS67ihXJgRR9AYsw0N1NLVy/HqFumxDQCtnb0onElTJrqHzOEzzdgdWoIt4bfS46YTHxnqkf229pY7AzbpwQ4MS5JmAEx4KuFTT0FaGlxyyTrS0pw/G9Enz5/D6cZO8gf5TvjFm0V9/5ess0IYi6GCLaXUdOBa4C6tdZvWejvwMnDDOJ5uPc41ab/VWndrrX+PMyHbJZ4qr6fkpFr40+dzMCn4yHmJ47oQKKiwojVckCY9tv6uf+a1iX5puveFW5YsF5fCPymlyHIlESoon9joVn55I/NmRkiWtgCREhNOdHgwh86Mf//Lp56CW2+FigrQWlFR4fzZiAHXR85LZFqwmef3fXgq4RtHqtl/qgmzbMQrhCEpT24WOVFKqfOBHVrr8H7Hvgus01p/fJDz7wG+DdiBs8D9WusHXfd9G7hca31lv/NfAbZorf9vkOe6FbgVID4+Pufvf/+7J6s2Kvfv7+J4o51frw8nxDy29TX/LO7h9bJe/nhpOKFBira2NiIiBl9E6y8CoQ4w9nqctNr52d4ubA4wK/jhBWHMG+cm1b/f10Vlm4Ofr53YYulAeC0Gq8OGDRsKtNYrRvN4I7QRnuQvr+lJq5379nRh1xBsgjtXfvB5GEsdHFrz9Xc7uCAxiC+eF+rNIo+Zv7wWw/FVHX61t4uWHs29a6aN6/HXXZdLTU3YOccTErp49tk8w7URfz7YxcE6O7/d4LxOqO90cPeOTmaGm7hufjAnrQ7mx5jH/Z3Rn7wvjSEQ6gCBUY9xX0dorQ1zAy4GqgccuwXYOsT5C4HZgBlYjTPg+pzrvruAZwec/xRwz0jlyMrK0r6w/USdTr3zFf38vtNjfuyn/7RTf+L+7X0/b9myxYMl841AqIPW46vHnrIGnf1fr+kbH80b9+91OBx6xU/e1t9+dv+4n8MtEF6LweoA5OtxtFW+aiM8yV9e0/s3n9Bzf/CKTr3zFZ125yv6/s0n+u4bSx2OnW3WqXe+ov+ZP/b21dv85bUYjq/q8Is3jun0H76qO3ts43q8UlrDuTelnPcbrY3YWlSrU+98Rb96qEr32uz6U3/coRfd/YYuq2vz+O+S96UxBEIdtA6Meoz3OmJSpxEqpbYqpfQQt+1AGzAwlV4UMOjqV631Ua11ldbarrXeCfwO+A/X3WN6LiNYnRFLetx0nso7NabHddvsHDjdxEpZhxAwVqbFcMWiRI5UtuBwjG/0ubKpk7rWblmvJfxabnosIUHOryqlGPf0qL19W2PIVOtAsiQpGrtDc/Rsy7gen5IytuO+tiYjlvjIUJ7fV8lv3immoMLK/37yPNLipvu6aEKIIUxqsKW1Xq+1VkPcLgKKgSClVGa/hy0FCkf7K6BvM/BCYIn68MYsS8bwXJNOKcX1q1LIr7ByvHr0XxxHKpvpsTlYIRcRAWV9djwN7T0cqRrf4u/9rvVa56dIEC78l3sfnAWzIokMC2L5ODsPCsobmRkZSnLM+KabCWPqS5Jxenzrtv73fyF8wCzr8HDncSMKMpu4eulsNh+v4YEtJVwyP56rl83xdbGEEMMwVIIMrXU78Dxwr1JqulJqDXA18LfBzldKXa2UsiinC4Bv4sxACLAV51qubyqlQpVSt7uOb/ZqJSbo2uVJhASZxjS6tbffZsYicKzNjEcp2FpUN67H7z/VRFiwiezESA+XTIjJlZNq4abVaTR32jhR2zau59hbbmVFmsUjGyML40iMCiM+MpRDlePrlNq0CR56CFJTQSlNaqrz502bPFxQD5o/KxL3hIedJxskzbsQBmeoYMvlNmAaUAs8A3xNa10IoJS6WCnV/5v2OuAkzqmBfwV+rrV+AkBr3QNcA9wINAFfAq5xHTcsy/QQPrZkFi/sr6S92zaqx+wtayQ9fjqxEcZa9C0mJjYilCVzZrC1qHZcj99/2sqSOdEEy75rIgCsznBu1LrzZP2YH1vV1EllUycrZH+tgKOUYsmcGRNK/75pE5SXw+bN71FebuxAC6CmpatvCk+vXdK8C2F0hrsK01o3aq2v0VpP11qnaK2f7nffNq11RL+fP6e1jtXO/bXma2d69/7PtV9rnaO1nqa1Xq613j+ZdRmvTatSaeu28fLBqhHPzS9vZPvJetJlvnZAWpc9kwOnm7C2j62PoNtmp7CyRdZriYCRHBNOkmUau8ZxYenel0jWawWmJUnRlNS10TbKDkp/l5seR2iwSdK8C+EnDBdsCVieEs38xEgeer+EB7acGHKKQEGFlU2P7Kbb5uC94jqZShCA1mfH49Dw/omxTSUsrGqhx+6Q9VoioKzOiCWvtBH7GJPG5Jc3Eh5iZsEsmVIbiJYkz0Br5/rlqcC9jvGOy7N56uZc2aRbCIOTYMuAlFKszYqnrL6DX71ZzKZH8gYNpP5VcIZumwMAu0PLVIIAtDQpGkt4MO+Ncd3WB8kxZGRLBI7VGXE0d/ZybIyZ5/aWW1meYiFIptQGpCVzXEkyJrC5sb/JSbXw9Q3zJNASwg/IN49BhQc7NyTUQFevg1cPn+27z+HQ/Om9Ep7dewoFmBSEyFSCgGQ2OQPv94rrxpQCfv8pK3Oip5EQde5mnUL4qwsznG3czpLRr9tq6erleHULKySBUMCKjQhlTvS0Ca3bEkIIbwnydQHE4C7OiudP75fQ3etAA0/sLGNasIklSdH8/PXjlNa389HFs7jugmQOnWkmNz1WergC1PrseF46UMWRqmaWJI1upGr/qSaWyaiWCDAJUWFkxE9nZ0kDt67NGNVj/pF/Gq1hRliwl0snfGlJ0sSSZAghhLdIsGVQ7jnZeaUNLJgVyauHqnlgS0nf/UEmxZfWpJGTFsPFmfE+LKnwtv4p4EcTbNW2dFHZ1MkX16R5v3BCTLLVGXH8a98Zeu2OEc8tqLDy09eOA/DzN46zJDlaOqUC1JKkaF4/Uk1TRw/R4SG+Lo4QQvSRaYQG5p6Tfcn8BP7vM0u5bmVy331aa/LKGn1YOjFZxpoCfv9p2cxYBK7VGbF09NhHtT7nhf2Vfck0JEV2YFvq3txYRreEEAYjwZYf+fSKZMIk3euUtD57JvtHmQL+9cPVmBT02OyTUDIhJpe73dt5cvjAqdfuYJsri6e0mYFvkStJxsPvl0pmXiGEoUiw5Uck3evUtT47Hj2KFPB7yhp46UAlDg1ffHyvXHSIgGOZHsLCWVHsLBk+2HpsexkVDR18/4psaTOngJO1bShg28n6ITP4CiGEL8iaLT+Tk2qRC4YpaElSNJGhZh56r5QkS/iQ74EH3yvFnbOw1+acNiXvFxFoVmfE8te8CnoyBs+2ebqxg9+8U8zGBQl8bX0GSqlJLqGYbP2niPZI2yeEMBAZ2RLCDxw43UR7j53Csy1c//DgvbbVzV3sOlmPScm0KRHYVs+Lpcfm4GTTuUkytNbc9dIRzEpx79WLJNCaInLTYwkNdl7SaA3nJ0s2ViGEMUiwJYQf6N9r221zsK343OmE//PKUTRw/+eWy7QpEdBWpsVgNimONXx4XWJBhZVvPrOfrUV1fOfybGZHT/NRCcVkc0+zv/6CFDSwdZA2UgghfEGmEQrhB3LTYwkJMtFjc+DQsKu0nm86MjGZnL32W4tqefXwWb57eRZXLZnl49IK4V2RYcFkxE9nR1U7BRVWlqdE835xHbf8rYAemwOlYLErO52YOtzT7DXw6PYyrl42m0Wzz30fFJQ3squ0gQsz4qRDSgjhdRJsCeEH+u+7Vt/WzV92lPPbd4q54/Jsunrt3P1SIenx07llbbqviyqE1xVUWCmta8fm0PzHgzsJCzbT2fvBKJcC9pQ1sjItxneFFD7zg4/M5+2j1fzohSM8/7XVmE0fTCV9bu8pfvD8YdAQGnxSZgAIIbxOgi0h/ERfr63WtHfb+P3mk2QlRlJc3cqpxg6evnkVoUFmXxdTCK/LK23AoZ2pYDSwcHYki+fM4Kndp7A7NCGyXnFKmxEezF0fW8i3nj3AU7sruPHCNM5YO/jlm0W8dKCq7zxJIiSEmAwSbAnhZ5RS/M8151FS184dzx3A7oC1mXGsnhfn66IJMSn6ptX2OggJNvGjqxaSk2rh40vnkFfaQG56rFxAT3GfWDqbfxac4b7Xjk39k1QAAAzuSURBVPPygSoOnGnCrBTXLp/Dq4fO0mt3SBIhIcSkkGBLCD8UGmTmtvUZfPmJfAB2lzVSUGGVC0wxJbin1T7zzl4+t3Fl3/tetsYQbkoprluZzLYT9eRXWDErxe8/fz5XLErk+lWpEpQLISaNBFtC+Knj1a2YFDg02OwyHUZMLTmpFlozQuQ9L4ZU3tDR10aC5mRtG1cskqBcCDG5JPW7EH7KPZVK9tQSQohzSRsphDACw41sKaVigEeBy4F64Ida66eHOPd14OJ+h0KAIq31Ytf95UAC4E5TtVNrfbmXii7EpOqfoVCmwwghxIdJGymEMALDBVvAA0APziBpGfCqUuqg1rpw4Ila6yv7/6yU2gpsHnDax7XW73iprEL4lEyHEUKIoUkbKYTwNUNNI1RKTQeuBe7SWrdprbcDLwM3jOKxaThHuf7qzTIKIYQQQgghxGgo7dqrxAiUUucDO7TW4f2OfRdYp7X++AiPvRu4RGu9vt+xcmAazqByP/A9rfXBIR5/K3ArQHx8fM7f//73iVXGx9ra2oiIiPB1MSYkEOoAgVGPQK3Dhg0bCrTWK0bzeGkjjCcQ6gCBUY9ArYO0EYH3mvqbQKgDBEY9xt1GaK0Nc8M5MlU94NgtwNZRPPYkcNOAY2twBlvhwA+BaiB6pOfKysrS/m7Lli2+LsKEBUIdtA6MegRqHYB8PY62StoIYwiEOmgdGPUI1DpIG+HfpA7GEQj1GG8bManTCJVSW5VSeojbdqANiBrwsCigdYTnvQhIBP7Z/7jWeofWulNr3aG1vg9o4sMJNYQQQgghhBDCKyY1QYbuN8VvMK41W0FKqUyt9QnX4aXAOckxBvgC8LzWum2kIgBqNGUVQgghhBBCiIkwVIIMrXU78Dxwr1JqulJqDXA18LehHqOUmgZ8Bnh8wPEUpdQapVSIUipMKfU9IA7Y4bUKCCGEEEIIIYSLoYItl9twrrOqBZ4BvqZdad+VUhcrpQaOXl2Dc3rglgHHI4EHAStQCXwEuFJr3eDFsgshhBBCCCEEYMB9trTWjTgDqMHu2wZEDDj2DM6gbOC5hcASb5RRCCGEEEIIIUZixJEtIYQQQgghhPB7EmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBcYLthSSt2ulMpXSnUrpR4fxfnfVkpVK6ValFKPKaVC+92XppTaopTqUEodV0pt9GrhhRBCCCGEEMLFcMEWUAX8BHhspBOVUlcAPwAuBVKBdOC/+53yDLAfiAX+H/BPpVS8pwsshBBCCCGEEAMZLtjSWj+vtX4RaBjF6V8AHtVaF2qtrcD/ADcBKKWygOXAj7XWnVrrfwGHgWu9U3IhhBBCCCGE+ECQrwswQYuAl/r9fBBIUErFuu4r1Vq3Drh/0WBPpJS6FbjV9WO3UuqIF8o7meKAel8XYoICoQ4QGPUI1DqkjvbB0kYYUiDUAQKjHoFaB2kj/JvUwTgCoR7jaiP8PdiKAJr7/ez+f+Qg97nvnzPYE2mtHwIeAlBK5WutV3i2qJNL6mAcgVAPqYO0EUYUCHWAwKiH1EHaCCOSOhhHINRjvHWY1GmESqmtSik9xG37OJ6yDYjq97P7/62D3Oe+vxUhhBBCCCGE8LJJDba01uu11mqI20XjeMpCYGm/n5cCNVrrBtd96UqpyAH3F46/BkIIIYQQQggxOoZLkKGUClJKhQFmwKyUClNKDTXd8a/Al5VSC5VS0cB/AY8DaK2LgQPAj13P8UlgCfCvURTjoYnWwwCkDsYRCPWQOnjvuXxF6mAcgVAPqYP3nstXpA7GEAh1gMCox7jqoLTWni7IhCil7gF+PODwf2ut71FKpQBHgYVa61Ou8+8A7gSm4Qykvqq17nbdl4Yz+FoFnAK+rrV+x/u1EEIIIYQQQkx1hgu2hBBCCCGEECIQGG4aoRBCCCGEEEIEAgm2hBBCCCGEEMILJNjqRykVo5R6QSnVrpSqUEpd7+syjUQpdbtSKl8p1a2UenzAfZcqpY4rpTqUUluUUqPenHEyKaVClVKPuv7mrUqpA0qpK/vd7y/1eFIpdVYp1aKUKlZK3dzvPr+og5tSKlMp1aWUerLfsetdr1G7UupFpVSML8s4FNcWE11KqTbXrajffROqg7QRviFthPFIGzHkc0sb4QPSRhiPtBEfkGDrwx4AeoAEYBPwoFJqkW+LNKIq4CfAY/0PKqXigOeBu4AYIB94btJLNzpBwGlgHTADZ1bJvyul0vysHvcBaVrrKOATwE+UUjl+Vge3B4C97h9cn4M/Azfg/Hx0AH/0TdFG5XatdYTrlg0eq4O0Eb4hbYTxSBsxOGkjfEPaCOORNsJNay03Z5KQ6TgbyKx+x/4G/MzXZRtl+X8CPN7v51uBnQPq1wnM93VZR1mfQ8C1/loPIBs4C3zG3+oAXAf8HbgHeNJ17KfA0/3OyXB9XiJ9Xd5Byr8VuHmQ4xOqg7QRxrpJG+HTsksbMfjzShthoJu0ET4tu7QR/W4ysvWBLMCmnftzuR0EjN4jNZRFOMsPgNa6HSjBD+qjlErA+XoU4mf1UEr9USnVARzH2Ui+hh/VQSkVBdwL3DHgroF1KMF1UTF5pRuT+5RS9UqpHUqp9a5jE62DtBEGIW2E70gbMSxpIwxC2gjfkTbiXBJsfSACaBlwrBmI9EFZPCECZ/n7M3x9lFLBwFPAE1rr4/hZPbTWt+Es28U4h/y78a86/A/wqNb6zIDj/lSHO4F0YA7ODQj/rZTKYOJ1kDbCAKSN8DlpI4YmbYQBSBvhc9JGDCDB1gfagKgBx6KAVh+UxRP8rj5KKRPOKRc9wO2uw35XD621XWu9HUgCvoaf1EEptQzYCPxmkLv9og4AWuvdWutWrXW31voJYAdwFROvg9/8DUbJ7+ojbYRvSRsxIr/5G4yS39VH2gjfkjZicEHeKaZfKgaClFKZWusTrmNLcQ5B+6NC4AvuH5RS03HOLTVkfZRSCngU54LDq7TWva67/KoeAwTxQVn9oQ7rgTTglPPlIAIwK6UWAm/g/DwAoJRKB0Jxfm6MTgMK5997InWQNsKHpI0whPVIGzEcaSN8SNoIQ1iPtBGDPNoAC9GMcgOeBZ7BufBwDc6hwUW+LtcIZQ4CwnBmsPmb6/9BQLyr/Ne6jv0cyPN1eYepx5+APCBiwHG/qAcwE+eC0AjADFwBtOPMJuQvdQgHEvvdfgX801X+RTinx1zs+nw8CTzr6zIPUodo19/e/TnY5HodsjxRB2kjfFoPaSN8XwdpI0Z+fmkjfFcPaSN8XwdpIwZ7Tl9Xykg3nOk0X3T9UU8B1/u6TKMo8z04I+7+t3tc923EucCyE2dmlTRfl3eIOqS6yt2Fc4jWfdvkL/VwNSTvAU2uD+Jh4JZ+9xu+DkO8t57s9/P1rs9FO/ASEOPrMg7xOuzFOaTf5PrivcxTdZA2wmd1kDbCgDdpIwZ9fmkjfFMHaSMMeJM2wnlTrgcKIYQQQgghhPAgSZAhhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsiSlPKRWllLpHKbXA12URQhiPtBFCiOFIGyGGI8GWELAC+DEQ7OuCCCEMSdoIIcRwpI0QQ5JgSwg4H+gGjvq6IEIIQ5I2QggxHGkjxJCU1trXZRDCZ5RSx4D5Aw4/r7W+1hflEUIYi7QRQojhSBshRiLBlpjSlFIrgWeBQuCnrsNntdYVviuVEMIopI0QQgxH2ggxkiBfF0AIHzsIJAF/0Frn+bowQgjDkTZCCDEcaSPEsGTNlpjqFgEhwD5fF0QIYUjSRgghhiNthBiWBFtiqlsOaOCArwsihDAkaSOEEMORNkIMS4ItMdWdD5RorVt8XRAhhCFJGyGEGI60EWJYEmyJqW4hkqpVCDE0aSOEEMORNkIMSxJkiKmuCViulLoCaAZOaK0bfFwmIYRxSBshhBiOtBFiWJL6XUxpSqnzgEeBJUAYcLHWertvSyWEMAppI4QQw5E2QoxEgi0hhBBCCCGE8AJZsyWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBf8f8gc3XacdgyWAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "def plot_series(series, y=None, y_pred=None, x_label=\"$t$\", y_label=\"$x(t)$\", legend=True):\n", " plt.plot(series, \".-\")\n", " if y is not None:\n", " plt.plot(n_steps, y, \"bo\", label=\"Target\")\n", " if y_pred is not None:\n", " plt.plot(n_steps, y_pred, \"rx\", markersize=10, label=\"Prediction\")\n", " plt.grid(True)\n", " if x_label:\n", " plt.xlabel(x_label, fontsize=16)\n", " if y_label:\n", " plt.ylabel(y_label, fontsize=16, rotation=0)\n", " plt.hlines(0, 0, 100, linewidth=1)\n", " plt.axis([0, n_steps + 1, -1, 1])\n", " if legend and (y or y_pred):\n", " plt.legend(fontsize=14, loc=\"upper left\")\n", "\n", "fig, axes = plt.subplots(nrows=1, ncols=3, sharey=True, figsize=(12, 4))\n", "for col in range(3):\n", " plt.sca(axes[col])\n", " plot_series(X_valid[col, :, 0], y_valid[col, 0],\n", " y_label=(\"$x(t)$\" if col==0 else None),\n", " legend=(col == 0))\n", "save_fig(\"time_series_plot\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "mVW_UPLp5jk5" }, "source": [ "## Computing Some Baselines" ] }, { "cell_type": "markdown", "metadata": { "id": "opK47Oz95jk5" }, "source": [ "Before we start using RNNs, it is often a good idea to have a few baseline metrics, or else we may end up thinking our model works great when in fact it is doing worse than basic models. The simplest approach is to predict the last value in each series. This is called *naive forecasting*" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "zQIzWhQQ5jk5", "outputId": "be199c4b-cff9-45d6-91bd-ba16b2191cd9", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.020211367" ] }, "metadata": {}, "execution_count": 7 } ], "source": [ "y_pred = X_valid[:, -1]\n", "np.mean(keras.losses.mean_squared_error(y_valid, y_pred))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "id": "4y1kxSFw5jk6", "outputId": "0f0b9c0d-9df8-46ae-917e-4bb3610496cf", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVzUdf7A8dd7OAVUEAEPFLzA+8ID80JTOy23w0q7ti13O7Z2267ddstta2u7tj3Ksl9tW1mtlnZumibkkReoqCheCCoiyCHIJTDz+f0xAwsIyDHHd+DzfDy+D53P9/v9zGe+zMx7vp9TlFJomqZpmqOZXF0ATdM0rWPQAUfTNE1zCh1wNE3TNKfQAUfTNE1zCh1wNE3TNKfQAUfTNE1zCh1wNE3TNKcwZMARkQdEJFFEzovIexc59tciclpEikTkXRHxqbUvUkTiRaRURFJFZJbDC69pmqY1yJABBzgFPAu829RBInIZ8ARwKRAB9Af+WOuQj4FdQDDwJPCpiIQ4osCapmla08TIMw2IyLNAuFLqzkb2fwSkK6V+Z3t8KbBMKdVDRKKAvUB3pdQ52/6Ntv1vOuUFaJqmaTU8XV2ANhoGfFHrcTIQJiLBtn1p1cGm1v5hDWUkIouARQC+vr4xffv2dUyJ3YjFYsFkMupNsPPo66CvQTV9HZq+BocOHcpVSjVai+TuAScAKKz1uPr/nRvYV72/d0MZKaWWAksBoqOj1cGDB+1bUjeUkJBAXFycq4vhcvo66GtQTV+Hpq+BiGQ0da67h+pioEutx9X/P9fAvur959A0TdOczt0DTgowqtbjUUC2UirPtq+/iHSutz/FieXTNE3TbAwZcETEU0R8AQ/AQ0R8RaSh6r/3gZ+JyFARCQR+D7wHoJQ6BOwGnrad/xNgJPCZU16EpmmaVochAw7WwFGGtcvzrbb//15E+opIsYj0BVBKrQZeBOKB40AG8HStfG4GxgEFwAvADUqpM057FZqmaVoNQ3YaUEotBhY3sjug3rGvAq82kk86EGe/kmmapmmtZciAY3RFRUXk5ORQWVnp6qI4VNeuXTlw4ICri+E0/v7+hIeHd/hur5rmKDrgtFBRURHZ2dn07t2bTp06ISKuLpLDnDt3js6dO1/8wHbAYrGQmZlJbm4uoaGhri6OprVL+qdcC+Xk5NC7d2/8/PzadbDpaEwmE2FhYRQW1h+6pWmaveiA00KVlZV06tTJ1cXQHMDLy4uqqipXF0PT2i0dcFpB39m0T/rvqmmOpQOOpmma5hQ64GiapmlOoQOOpmma5hQ64HQAItLkduedd7qsbJGRkbz88ssue35N05xHBxwXWbYMIiPBZLL+u2yZ454rKyurZnv77bcvSPvb3/7WovwqKiocUUxN09o5HXBcYNkyWLQIMjJAKeu/ixY5Luj06NGjZgsMDKyTVlJSwu23306PHj3w9/dn7NixfP3113XOj4yMZPHixdx1110EBgaycOFCAN5991369u2Ln58fc+fO5Y033rigp9dXX31FTEwMvr6+9OvXjyeffLImYMXFxZGRkcGjjz5ac7elaVr7pQOOCzz5JJSW1k0rLbWmO1txcTFXXHEFa9euJTk5meuvv57rrruO1NTUOse9+uqrDB48mMTERP785z+zZcsW7r77bu6//352797NNddcw9NPP13nnDVr1rBw4UIeeOABUlJSePfdd/n000/53e9+B8DKlSsJDw/nqaeeqrnb0jStHVNK6a3eFhUVpRqzf//+Rvc1l4hS1nubuptIm7O+qBUrVijrn71xEydOVH/6059UUVGRUkqpiIgIdfXVV9c55uabb1aXXXZZnbR77rmnTt5Tp05VzzzzTJ1jVq1apfz9/ZXFYqnJ+6WXXmr167G3hv6+8fHxzi+IwehrYKWvQ9PXAEhUTXy36jscF+jbt2XpjlRSUsJjjz3G0KFDCQoKIiAggMTERI4fP17nuHHjxtV5nJqayoQJE+qkTZw4sc7jpKQknnvuOQICAmq2BQsWUFJSwunTpx3zgjRNMyw9eacLPPectc2mdrWan5813dkeeeQRVq9ezcsvv8ygQYPw8/Pj9ttvv6BjgL+/f4vztlgsPP3009x4440X7AsJCWl1mTVNc0864LiArc2dJ5+E48etdzbPPfe/dGfatGkTt99+O9dffz0A5eXlHD16lKioqCbPGzx4MDt27KiTtn379jqPx44dS2pqKgMHDmw0H29vb8xmcytLr2maO9EBx0UWLnRNgKkvKiqKVatWce211+Ll5cUf//hHysvLL3regw8+yJQpU3jppZeYN28eGzZsYNWqVXWOeeqpp7j66quJiIhg/vz5eHp6sm/fPrZv386LL74IWHvAbdy4kVtvvRUfHx+6d+/ukNepaZrrGbYNR0S6icgqESkRkQwRWdDIcd/alp2u3ipEZG+t/ekiUlZr/3fOexXG9+qrrxIaGsrUqVO54ooriI2NZerUqRc9b9KkSbz99tv8/e9/Z+TIkXz++ec8/vjj+Pr61hxz2WWX8c033xAfH8+ECROYMGECL7zwAn1rNVY988wznDhxggEDBuhqNk1r54x8h/M6UAGEAaOBb0QkWSmVUvsgpdQVtR+LSAKwvl5ec5VS6xxYVrdxww03YO1MYhUREcG6dXUvzSOPPAJYF2ADSE9PbzCvu+66i7vuuqvm8a9//esLqs/mzJnDnDlzGi1PbGwsycnJLXoNmqa5J0MGHBHxB64HhiulioFNIvIlcBvwRBPnRQJTgTsdX0rtpZdeYvbs2QQEBLBu3TrefPNN/vznP7u6WJqmGZQhAw4QBVQppQ7VSksGpl/kvNuBjUqp9Hrpy0TEBOwCHlVK6Z/UdpCYmMjLL79MYWEh/fr14/nnn+ehhx5ydbE0TTMoqV29YhQiMhVYoZTqUSvtHmChUiquifOOAM8qpd6rlTYZ2AkI8JBtG6yUOlvv3EXAIoCQkJCY5cuXN/gcXbt2bbLXVXtiNpvx8PBwdTGc6siRIxcsM11cXExAQECdtMMFVRzIszA02IOBQe3/GjV0DToifR2avgYzZsxIUkqNa3Anxr3DKQa61EvrApxr7AQRmQL0AD6tna6U2lzr4fMicgfWarev6h23FFgKEB0dreLi4hp8ngMHDtC5c+dmvQh3d+7cuQ7zWqv5+voyZsyYOmkJCQnUfj8kZRTw/JofsSj4b4aZZXfHEhMR5OSSOlf9a9BR6evQtmtg1F5qhwBPERlUK20UkNLI8QB3ACttbT5NUVjvdjStVVbtPInFVjFwvtLC1rQ81xZI09yEIQOOUqoEWAk8IyL+tmqxa4EPGjpeRDoB84H36qX3FZHJIuItIr4i8ijQHdjcQDaa1izJJ/9XG6uAIT071l2gprWWIQOOzX1AJyAH+Bi4VymVIiJTRaT+Xcw84CwQXy+9M7AEKAAygcuBK5RS+iep1iqbj+SyN7OIn06O5K4p/fAQ+C4l29XF0jS3YNQ2HJRS+VgDSf30jUBAvbSPsQal+semACMdVUatY1FK8dKag/Ts6svjlw/G18sDE/DO5mPcGhvB8N5dXV1ETTM0I9/haJqhrDuQw+4TZ3no0kH4ell7pj04axDd/LxZ/GUKRuzxqWlGogOOM734IsTXr/WrJz7eepyb+vTTT+us3Pnee++1uRtpQkICIkJubm5bi9dqFovi5TUH6dfdn+tjwmvSu/h68ehl0SRmFPBl8imXlU/T3IEOOM40fjzMn9940ImPt+4fP97uT33nnXfWLOPs5eVF//79eeSRRygpKbH7c9V20003kZaW1uzjIyMjefnll+ukXXLJJWRlZREcHGzv4jXbV3tOcTD7HL+eHYWXR92PzY3j+jC8dxde+DaV0ooqF5VQ04xPBxxnmjEDli9vOOhUB5vly63HOcCsWbPIysoiLS2NZ599ljfeeKNm3rTaqqqq7FY91KlTJ0JDQ9uUh7e3Nz169Khz5+RMVRbFq2sPMbhHZ64e0fOC/R4mYfHcYWQVlvNmwlEXlFDT3IMOOM7WUNBxQrAB8PHxoUePHvTp04cFCxawcOFCPv/8cxYvXszw4cN57733GDBgAD4+PpSUlFBYWMiiRYsIDQ2lc+fOTJ8+ncTExDp5vv/++0RERODn58fVV19NdnbdHlsNVan997//ZeLEiXTq1Ing4GDmzp1LeXk5cXFxZGRk8Oijj9bcjUHDVWorV65kxIgR+Pj40KdPH5577rk6QTIyMpJnn32Wn//853Tp0oXw8HBeeumlVl23TZlVZOSV8uhl0ZhMDQe9cZHduGZUL97akMaJ/NIGj9G0jk4HHFeoHXSeesopwaYhnTp1orKyEoBjx47x0UcfsWLFCpKTk/Hx8eHGG28kMzOTr7/+ml27djFt2jRmzpxJVlYWANu2bePOO+9k0aJF7N69m7lz5/LUU081+ZyrV6/mmmuuYfbs2SQlJREfH8/06dOxWCysXLmS8PBwnnrqKbKysmqep76kpCRuvPFGrrvuOvbu3csLL7zA888/zz//+c86x/31r39lxIgR7Ny5k8cff5zHHnuMLVu2tOgalVea+eJIJWP7BjJzcNN3ar+9cjAmER77NJnX44+QlFHQoufStHZPKaW3eltUVJRqzP79+xvd12J/+INSYP3Xwe644w511VVX1Tzetm2bCg4OVvPnz1dPP/208vT0VKdPn67Z//333yt/f39VWlpaJ59Ro0apv/zlL0oppW655RY1a9asOvt/9rOfKevbyupf//qX8vf3r3l8ySWXqJtuuqnRckZERKiXXnqpTlp8fLwC1JkzZ5RSSi1YsEDNmDGjzjFPP/206t27d518br755jrHDBw4UP3pT39q9LmVuvDv+/QXe1XE41+r9zYfa/K8ak98mqwiHv9aRT7+tYr+/X9VYnp+s84zuvj4eFcXwRD0dWj6GgCJqonvVn2H4yrx8bBkCfzhD9Z/L9Z7zQ5Wr15NQEAAvr6+TJo0iWnTpvGPf/wDgPDwcMLCwmqOTUpKorS0lJCQEAICAmq2ffv2cfSotZ3iwIEDTJo0qc5z1H9c365du7j00kvb9DoOHDjA5MmT66RNmTKFzMxMioqKatJGjqw7BKtXr17k5OQ0+3k2HcnlvR8zAHj+2wPNumPp0dW6AJ2ietob1/Ws0zSjMezAz3atfpvNjBlOqVabNm0aS5cuxcvLi169euHl5VWzz9/fv86xFouF0NBQNm3adEE+XbrUn1fVOGp3LKj9+qr3WSyWZuf18bbjNf+vrLLOmXaxSTqnDAphScJRyqssKKwTfVZUWfD21L/tNE1/CpytoQ4CTfVesyM/Pz8GDhxIRETEBV/G9Y0dO5acnBxMJhMDBw6ss1X3OhsyZAhbt26tc179x/WNGTOG77//vtH93t7emM3mJvMYMmQImzfXnQ5v06ZNhIeH23V268Iya/uWCfDyNBHb/+LdsmMiglh2TyyPXhbFjePCWZ96htvf3cbZ0gq7lUvT7CEpPZ9/rD/c9J27nccO6oDjTE31RnNS0GmuWbNmERsby7XXXsu3337LsWPH2LJlC08//TQbN24E4MEHH2TdunU8//zzHD58mLfffptVq1Y1me+TTz7JihUr+P3vf8/+/ftJSUnhr3/9K6Wl1p5dkZGRbNy4kczMzEYHev7mN7/hhx9+YPHixRw6dIhly5bxyiuv8Nhjj9nt9VeZLezNLCQuKoTrBnm1aAmCmIgg7p8xiJduGMVrN41mZ8ZZfvLGj6SdudhE5ppmf0kZBTWdWLIKy/g06SR3vLuN69/cwivfHWL+m1v4Mjmz4ZPtPHZQBxxn2rGj6Wqz6qCzY4dzy9UAEeHTTz9l5syZ3HPPPURHRzN//nwOHjxIr169AIiNjeWdd95hyZIljBw5kpUrV7J48eIm873yyitZtWoV3377LWPGjGH69OnEx8djMlnfis888wwnTpxgwIABhISENJjH2LFjWbFiBZ999hnDhw/niSee4IknnuCBBx6w2+vfdeIshWWV3DiuD1cP8G71ejfzxvTmo3smUlhWydx/bOLxz/bo3mua0yRlFHDL0q28tOYgNyz5kUnPr+eRFclsP/a/96BZKR78eDc//yCRXcfrvTftPHbQkCt+ulp0dLQ6ePBgg/sOHDjAkCFDnFwi1+iIC7BV/33/sjqVtzeksfOp2ezcurnNi279d08W93+0EwX4epncbtE2vfCYlbtdh5fXHOSf8UdqHs+IDuGxywdTcr6KW9/ZRmWVBU8PE3NH9WTt/hwKyyqZ2K8bs4aGUVFlJrZ/d+v7tFZwSRAhTqkGg42IuOWKn5rmUvGpOYyLDKKLb9NtXc11LO9/UwhVNLMDgqa1VXUfGpOAt6eJB2YOYkhPa6efZXfHsjUtj9j+wcREBFF8vopPth/njfijbDuWD4Cv1xHrj6NadzqRV1wB337bqk5OukpN0+rJPFtG6ulzFx3o2RKx/YNreqqZRJrVAUHT2upITjFBfl48PDvqgrtqa1vjwJq0AB9P7p7anzsnR9YsiVzdOxOwBpd77yXygw/g3ntb1aNWBxxNqyc+1TpWx54BJyYiiI/unoiftweTBwbruxvN4corzfxw6AxXjOjJAzMHNfs9N3lgd3y8THhIvd6ZtrGD6bfd1uqxg7pKTdPqiU/NoU+3TgwIaduyCvXFRHbjkgHBpOU6doZuTQPYkpZHaYWZ2UPDLn5wLTERQRdUt9Vuw0kXIfKnP23V2EF9h9MKuqNF+2SdfgM2H81lZnSoQ2anHtM3iLQzJRSWVto9b02rbd3+bPy8PZjUiurbOtVtdhw7aNiAIyLdRGSViJSISIaILGjkuMUiUikixbW2/rX2jxaRJBEptf07ui3l8vLyoqysrC1ZaAZVWVlJuVlRXmkhzo7VabWN6RMIwO6TZx2Sv6aB9cfTugPZTBsUUrM6bavYeeygYQMO8DpQAYQBC4ElIjKskWP/o5QKqLWlAYiIN/AF8CEQBPwb+MKW3iqhoaFkZmZSWlqq73TaEYvFQnZ2NntzKvH1MrXqV2FzjAjviggXjnfQNDvam1lIdtH5FlenXcDOYwcN2YYjIv7A9cBwpVQxsElEvgRuA55oQVZxWF/ja7aZTP8uIo8AM4HVrSlb9Txip06dqpnav70qLy/H19fX1cVwGn9/f/61M4/JA7q37VdhEzr7ehEV2pldx/UdjuY46/ZnYxKY0dY79ebM3lE9H2QzGDLgAFFAlVLqUK20ZGB6I8fPFZF8IAv4p1JqiS19GLBH1b0V2WNLrxNwRGQRsAggJCSEhISENr8Id1dcXHzB4mnt2aliC+l5ZUwLM9f5+xcXF9v1/RDmdZ7EY+eIj4932SqmLWXva+Cu3OU6rNpRxsBAE3t2/Gj3vNtyDYwacAKAonpphUBDw96XA0uBbGAi8JmInFVKfWzLp7A5+SilltryITo6WrnTaGJHcbdR1W21dMNRIJWfXzOF3oGdatLtfR1O+x1nw8q9RAwfT38794RzlI72XmiMO1yHkwWlnFgdz++uHEzctAF2z78t18CobTjFQP058LsA5+ofqJTar5Q6pZQyK6V+BP4G3NDSfDRtfWoOg3t0rhNsHGF0X1vHgRO6Wk2zv3X7rcu8zx7aw8UluZBRA84hwFNEBtVKGwWkNONcBTUDZVOAkVK33mJkM/PROpCi8koS0wvaXufdDINCO+Pv7aHbcTSHWHcghwEh/vTr7n/xg53MkAFHKVUCrASeERF/EZkMXAt8UP9YEblWRILEagLwINaeaQAJgBl4UER8RKR6OuH1Dn8RmlvZeCiXKouy6+wCjfEwCaP6BLLrhO6pptlXUXklW9PymNXW3mkOYsiAY3Mf0AnIAT4G7lVKpYjIVBGpvbDIzcARrNVk7wN/UUr9G0ApVQHMA24HzgJ3AfNs6ZpWY31qDl07edWMk3G00X0CSc06R1lF04vNaVpL/HDwDFUWxewhxgw4Ru00gFIqH2uwqJ++EWtngOrHt1wkn11AjN0LqLUbFovih0M5TI8KwdPDOb/BxvQNosqi2HeqkPGR3ZzynFr7t3Z/NsH+3ozpa8y5+ox8h6NpTrE3s5Dc4gpmDG54wTdHGF0944Bux9HspNJsIf5gDjMHh+JhMmZ3ex1wtA5v2bYMAAL9Wj0BRYuFdPYhPKiTbsfR7GbHsXzOlVc5rP1m2TKIjISZM6cTGWl93FI64GgdWlJGASsSTwJw74dJTl3+eUzfIH2Ho9nN2gPZ+HiamDqou93zXrYMFi2CjAxQSsjIsD5uadDRAUfr0L7cnUn1NBR1FptygtF9AjlVWM7pwnKnPafWPimlWLs/mykDu+Pnbf+m+SefhNLSummlpdb0ltABR+vQzhSfB7hwsSknGFMzAFRXq2lts2pXJicLyhgU5piZK44fb1l6Y3TA0Tqs8kozm4/kMXlgdx6eE33BEryONrRnF7w8hF16xgGtDZIyCnjs0z0A/GtzukOqhfv2bVl6Y3TA0TqstfuzKSyr5BfT+9dZ291ZfL08GNqrq55xQGuTrWl5VFmsFcNVZsdUCz/3HPj51U3z87Omt4QOOFqHtTzxBL0DO3HJAPs3sjbXmD6B7D1ZSJXZ4rIyaO5tQj/rOC7BcdXCCxfC0qUQEQEiiogI6+OFC1uWjw44WoeUebaMTUdyuT4m3KVjFsb0DaSs0szBbD2frNY6ft7WtZuuHNHTodXCCxdCejqsX/8D6ektDzagA47WQX2WdBKl4MaYcJeWY0wf65eDrlbTWqu6zeaJKwY7vVq4pXTA0Toci0WxPPEEkwcG06eb38VPcKA+3TrRzd9bL1WgtVpiegFhXawDiY1OBxytw9malsfJgjLmj+vj6qIgIozpE8iu47prtNY6SRkFjIvo5harx+qAo3U4yxNP0NnXk8uGGWOBqtF9Ajl6poTCskpXF0VzM1mFZWSeLTN8VVo1HXC0DqWwrJJv953m2tG98PXycHVxAGpm9n326/1OnVpHc3+J6db3y7hIHXA0zXC+TD7F+SoLN41r4Yg1B1K2yXU+TTrJwv/bqoOO1mxJGQV08vJgSM8uri5Ks+iAo3UoKxJPMLhHZ4b3Ns4HdM/JQsC6Nrqz53PT3FtiRj6j+wTi5aR1nNrKPUqpaXZwIKuIPScLmT+uj6EaWGP7B+NpGwvk6eHc+dw091V8vor9p4oY7ybVaWDggCMi3URklYiUiEiGiCxo5LhHRWSfiJwTkWMi8mi9/ekiUiYixbbtO+e8As1olieewMtDmDemt6uLUkdMRBCvzh8FwG2TItymAVhzrd3Hz2JREONGK8YaNuAArwMVQBiwEFgiIsMaOE6A24Eg4HLgARG5ud4xc5VSAbZtjiMLrdlPUkYBr8cfsUubxta0XD7efpzxEd3o5u+8hdaa65rRvekf4k9qlp5xQGuexIx8RP4367g7MGTAERF/4HrgD0qpYqXUJuBL4Lb6xyqlXlRK7VRKVSmlDgJfAJOdW2LN3ral5XHTW1t45buDbW5I/3ZvFgve3kZ5pYXEjHzDNsrPGdqDrWl5FJbq7tHaxSVlFBAd1pkuvl6uLkqziVLq4kc5mYiMATYrpfxqpT0CTFdKzW3iPAF2Am8ppd60paUDnbAG113Ao0qp5AbOXQQsAggJCYlZvny5/V6QmyouLiYgwD7raxwpMJOab2ZwNw8GBjXdHbnKovjD5jKySqzvTRNw3SAvrh7QsjuT4grFV0crWJtRRfXUmK3Jy57XoSlHzpp5dms5i0b6cEkv+y+i1RbOugZGZ5TrYFGK+9aVckkvT24f5uPU527qGsyYMSNJKTWusXON9a7+nwCgqF5aIdD5Iuctxvqd8q9aaQuxBiEBHgLWiMhgpVSduUSUUkuBpQDR0dEqLi6utWVvNxISErDHddh+LI8XvtuGRSm8Pc1NTjBYabZw/7KdZJWU4mESzBaFiHDLrPHNattIyihg0+Ez5Jw7z5fJpyg5X0Xc4BA2H8mjymzBy9PU7Lyq2es6XMw0i+KtlO85qYKIi4tx+PO1hLOugdEZ5TqknCqkfM0mrrlkOHFObpNsyzUwasApBur3W+0CNFrBLSIPYG3LmaqUOl+drpTaXOuw50XkDmAq8JX9iqs15Z1Nx2rW66iosrA1LbfBL/xKs4VffrSL7/Zns3juUEaEB/LSmlS2puVzqhmjqZMyCrh56RYqzdbnGhcRxJ+vG0FUWGeSMgrYmpZHbP9gwzbKm0zC7KFhfL4rk/JKs2EGpmrGU10tbNT3cmMM2YYDHAI8RWRQrbRRQEpDB4vIXcATwKVKqZMXyVthvdvRnMBsUew5WVhzwS0KNh/J42xpRZ3jKs0WHvx4F6tTTvPU1UO5c3I/YiKC+OBnExnbN5DfrdzL8bzSC5+gOl+L4i+rD9QEG5PAjMGhRIVZb4pjIoJcsshaS80ZGkZphZkfj+a6uiiagbnThJ21GTLgKKVKgJXAMyLiLyKTgWuBD+ofKyILgT8Ds5VSafX29RWRySLiLSK+ti7T3YHN9fPRHGPt/tNkFZbz69lRPDInigUT+7L9WD6XvbaBHw6dAayrFP7qk918u+80v79qCHdN6VdzvpeHib/dPAYEfvnJLiobWKisosrCw8t3s/1YAR4mwUPA20ELUTnapAHBBPh48l1KtquLohmYO03YWZtRq9QA7gPeBXKAPOBepVSKiEwFvlVKVbdaPQsEAztqXfwPlVK/wNrmswQYAJQDu4ErlFJ6KLcTKKVY8kMafbv5cf+MgTULnS2Y0Jdf/2c3d7y7nSuG9+BYbgmpp8/x+6uGcPfU/hfk06ebH3+5fiT3LdvJy98d5LdXDKnZd668kl98mMTmI3k8elk0sf26sfVYvqGrzpri4+lBXHQI6w5kY7Yoly4OpxnTqbPWCTvvntrv4gcbjGEDjlIqH5jXQPpGrJ0Kqh83etWVUinASIcUULuobcfyST5xlj/NG17ni3N476589cspPLIima/3ZAHgaZKaSSwbcuWIntwyoS9v/ZDG5AHdmRYVQnZROXe8u50jOcW8cuMorrctpuZOA+EaMntoGF/vyWL3iQJiItz7tWj2l2hrvxnnhu8NQ1apae3DmxzhBScAACAASURBVD8cpXuAd4OravraJhysjkNKqYvOIfbU1UOJCgvglx/v5PHPkrnq7xs5kV/Ku3eOrwk27cGMwaF4eYiuVtMalJSej5+3B0N6XqzTrvHogKM5xIGsIhIOnuHOSyIb7W0V2z8Yb08THgJezWhz6eTtwX1xAyksq+I/O06SW1zB4muGMS0qxBEvwWW6+HoR2z+YNSmnMeI4Oc21EjMKGN0nEE83mbCzNvcrseYWlm5Iw8/bg1tjIxo9JiYiiGV3x/LwnOgmx+bUlnm2rKbHm0kg59z5Jo93V3OG9SA9r5QjOcWuLopmIMXnqziQVcQ4N2yfBB1wNAc4WVDKl8mnuGVCXwL9mh7R39LuyrH9g/HxMrl1T7TmmD0kDIDv9utqNe1/3HHCztoM22lAc1/vbDqGAD+bYv9eNNV3RUYfxNlWPbr6MqpPIN+lnOb+GQNdXRzNINxxws7a7H6HIyJ/F5GvG0jvIiKLRWRIrbRficheEdF3Wu1EQUkFn2w/wTWje9Er0DGD0txlEGdbzRkaRvLJQk4Xlru6KJpBxKfm0N3fm8PZ7lnVatcvehEZAPwC65xm9Y0DngZqT236FhAC3GHPcmiu88HWDMoqzfx82gBXF8XtXTbMWq229oCuVtNgR3o+yScLOVNc4bZLkdv7zuJXQLJSKrGBfWOA88D+6gSlVBnwPvCIncuhucCWo7ksSThKTEQQ0T3cr8um0QwICaBXV1/e2Zjmll8umn19viuz5v/uuhR5swKOiAwUkUoReaZe+hLbSpvjRMQHuBX4qIHzDwAvAz5ApYgoEfnMtvsTYKiIXNKmV6K5VFJGAbe9s52ySjN7TxbqL0g72Hn8LNnnzpOeV8qCt93zF61mP6UVZoBmDyMwomYFHKXUEeD/gF+JSDCAiDwF3AX8xHZHEwsEAhsbyOJ2IA3rDM2TbNvDtn27sc4CfXnrX4bmalvT8mpmhDZb3PPXl9FsTcurGYdzvsrCdymnXVwizZX2ZhYyoneXFg0jMJqWVKk9A3gAT4jI3VjbY25TSq2z7Y/FOhPzngbOTQbCgfVKqa22LQNAKWWx7Y9t5WvQDKB61LPgvr++jKZ6YGz1bAxfJZ8ir7h9jjvSmnYst4QjOcVcNzbcrTvMNLtbtFIqS0ReA35jO+9BpVTtZTF7AUVKqYoGTh8GeGNdCK0hZ4Co5pZFM55z5VUA3Bobwbwxvd32A2EktbuAd+3kxZ++3s9P39vBR/fEEuCjRzR0JOts47Fm2cZnuauWvmsPY22H2aSUer3ePl+snQIaMhbr3c/uRvaXYV0GWnNTGw/nEujnxeJrhukZju0oJiKoJnj3CvTlnveTWPR+Iv/66Xh8PPUCbR3F2gPZDO7RmT7d/FxdlDZpdpWaiFyKtRvzFmCyiNSfhTkPaxtOQ8YAR5VS9ZeNrtYN0CtOuSmlFJsO5zJ5QHcdbBxo5uAwXrphJD8ezeNXn+zGbNHzrHUE+SUVJKbnM2eoe9/dQPN7qY0FVmHtOBAHHAeer3dYKuAtIg1N2zuUWt2hG9APONicsmjGcySnmNNF5Uwd1N3VRWn3rhsbzu+vGsK3+07ziw+TeD3+sO691s6tT83BomD20B6uLkqbXTTgiMhA4FvgO+CXtjaaPwJXisi0WodusP07oYFszgKjROQyEYmt7ulmyz8Qa/vNhgbO09zAxsPWm9MpOuA4xd1T+/OTMb1Zuz+bl9Yc4qa3tvDx9uNUVFlXQ03KKOD1+CM6ELUT6/Zn06OLL8N7d3F1UdqsyTYcEemBNdAcABbaepSBdbDmY8ALwCUASql0EdkOzMW6PHRtTwHvAJ9jbeuZCmyy7bsKqMB6B6W5oY2Hz9Cvuz/hQe5dv+xOBob6I1gbRqssit+u3Msfv0phQIg/B08XY7YovDxM/P7qIbZ1hwSTwOHsc+w7VcSYPkGMCO+Kl4fgYRK8PEzszyoiJbOQSQO61+n0kZRRwNdHK+jcr0B3BnGy8kozGw6f4bqxvd1uOemGNBlwlFKngQvW/FVKmYEhF57BEuBvInK/Uqq01vH7gImNPM2twIr6yz6LSDesQWoO1vad3yqlGhpUKlgD3922pP8DnlC2AQwiMtqWzxCsgfNnSqnGOi9oLXS+yszWtHxuHNd+FkBzB7H9u+PjdYTKKgueHiYeunQgucWV/HfvqZrxUBVmC099kdLg+e9vyWgi90MMDPGnf0gAJpGa5a6/Tt/qtuM/3NWPR3MprTC3i+o0sP9s0R8CjwP3YZ1ZoEm2YDATa7fp+l7HeucTBowGvhGRZNuy0bUtwroU9SisP/jWAseAN0XEG/gCeA14A/g58IWIDGqk+7bWQjszzlJWaWbKQF2d5kyNzZp91cieLHx7KxVmC54mE7+7ajADQzpjVoovd2eycmcmCutaQleO6MmM6FDMFsW6A9ms3Z9NdTeEKosiI6+UjPySmgBWPZ2KDjjOs3Z/DgE+nsT2d8/lCOqza8BRSlWJyE+xdoNujh7AnbaZDGqIiD9wPTBcKVUMbBKRL4HbgCfq5XEH8IpS6qTt3FeAe4A3sXZw8ARes93x/F1EHsEa5Fa34iVq9Ww6cgYPkzBpgB7o6Wy1u0zXTlt2T8PLNwT4ePLN3iwqqyx4eZr46eR+NfsHhAaw4fCZmn2vzB9NTEQQSRkF3PL2ViqqLJhE9IBeJ7LYfghMjwppP13glVKG27B2oy6tl/YI8FUDxxYCE2s9Hgecs/3/18C39Y7/GvhNA/ksAhKBRD8/P4X1bklvF9l63PaqClv4F5eXQ2/N27x7DVZdYm9U3r0GN3ufd6/Bqvf9H6ied73u8vJ3pM27Z5SKePxr5T80zuVlacGW2NR3u1HXoQkA6o/ZKQQamoI4wLav9nEBtrad+vsazUcptVQpNU4pNS48PNzlQdcIW3x8fJP784vP49s7iifunOfysrryOrjTdj7zAIVblnM+80Cz953PPMCtMWF4h0Rw6HSRy19DR3kvPPuvL/AwCSd3rHH5627uNbgYowacYqB+H8AuWCf5vNixXYBiZX31LclHa6Efj+ahFEwdFOLqomgOdkkvTzxNwqdJJ11dlA5j7f5sJkR2u+gy7e7EqAHnEOApIoNqpY0CGupyk2Lb19BxKcBIqdufcGQj+WgttPHwGTr7ejIqvKuri6I5WBcfYcbgUFbuyqTKbLn4CVqbZOSVcCi7mFntYHaB2gwZcJRSJVjH8jwjIv4iMhm4FviggcPfBx4Wkd4i0gvr5KLv2fYlAGbgQRHxEZEHbOnrHVn+jkApxcbDuVwyIBhPD0O+jTQ7uzEmnDPnzvPDoTOuLkq7t9Y2WWd7mM6mNiN/U9yHdULPHOBj4F6lVIqITBWR2gt6v4V1nZ29wD7gG1saytr1eR7W9XjOYl2/Z57SXaLb7FhuCZlny5iiq9M6jBmDQwn292ZFoq5Wc7R17WSyzvoMO8e5Uiofa7Con74Ra2eA6scK66wHjzWSzy4gxkHF7LA2HbFOZzNVj7/pMLw8TMwb05v3t6STX1JBN//207ZgJGdLK9iRXsC90we4uih2Z+Q7HM3ANhzKpU+3TkQEt69fYFrTbhwXTqVZ8fmuTFcXpd16d9MxzBZFn27tb8UWHXC0Fqs0W0ecTxkY0i7md9Kab3CPLozo3VX3VnOQpIwC/hlvHQf/9Jcp7W4CVh1wtBZLPnGW4vNVTNOzQ3dIN44Lt070ear+EDetrTYdPkP1MkfVUwm1JzrgaC224XAuJoFLBuiA0xFdM6oX3h4m3XnAAXy9rFPYmAS8PE3tbiohHXC0Ftt0+AwjwgPp6ufl6qJoLhDo583soWF8sTuzZg0ezT7SzpTg5+XBr2ZFtcuZuXXA0Vpkw6Ez7Dp+lkGh/q4uiuZCN4wLp6C0ku8PZLu6KO1G9azds4aG8eClg9pdsAEdcLQWSMoo4Gf/3oECvtyd1e4aNLXmmzYohLAuPqxoZ50HmlottXohOke973ceLyCvpII5w9rXYM/aDDsORzOerWl5VJmtLZpmi14bpSPzMAnXjQ3nzYSjvLg6lUuHhLn9eyEpo4CFb2/lfJUFTw/hoUsH0a+7dchf2pli/r7+MFVmxy1Et2bfabw9TMRFh9o1XyPRAUdrttj+wVSva9weGzS1lhnaszMKWJJwlHc3H3P7NoetabmU29qkKs2Kl7871OBxjliITinFd/uzmTwwmACf9vu13H5fmWZ3Q3t2wQSM69eNxy4f7NZfLlrbHc8vA6yLoLSH1UCLyqoA628qb08TL1w3kqG9uiAC+08V8einyVSaFeKAhegOZp/jeH4p98a1v9kFatMBR2u23SfOYlbw8+n93fqLRbOP2P7BeJgEs0Xh5eHed7wZeSV8sDWDEb27cNmwHkwa0L3OezwqzDqv2cMfbuX4OQvWMGs/36VkIwKXDmm/1WmgOw1oLZCUkQ/A2L462GjW5az/cPUQAB6a5b69qswWxSMrkvEQ4a3bxvHAzIZfS0xEEE9M9CW8Wyce+mQ358or7VaGNSmnGds3iNDOvnbL04h0wNGabUd6AVFhAe1qQSitbRZMiMDf24OTBWWuLkqrvbMpjR3pBSy+Zhi9Apuev6yTp/DaTWPIKizn6S/ss6zWyYJSUk4VcVk77p1WTQccrVnMFsXOjALGRXZzdVE0A/H2NHHJwO4kHDzTrCWGjebg6XO8vOYQc4aGcd3Y3s06JyYiiF/OHMjKXZl8sbvtk5hWr30ze2iPNudldDrgaM1y8PQ5zp2vYnyke1abaI4TFx1C5tkyjp4pvvjBBlJRZeHh5bvp7OvJn68b0aKJaB+YMZCxfQP5/ef7OFlQ2qZyfJeSTVRYAP26t//B1DrgaM2SaGu/GReh73C0uqrHjSQcdK+VQP+5/jApp4p47icj6B7g06JzPT1MvHbTGJSCh/+TjNnSuru7gpIKtqfnM6cD3N2ADjhaMyWmF9Cjiy/hQe1vjQ6tbXoHdmJQaIBbLT39nx3H+Uf8EaZHdefy4a37su8b7Mcz1w5je3o+d/5re6tmIPg+NQezRbXr2QVqM1zAEZFuIrJKREpEJENEFjRx7KMisk9EzonIMRF5tN7+dBEpE5Fi2/ad419B+5SYnk9MZJBe/0Zr0PSoELal5VNaUeXqolzU9mN5PPHZXpSCrWn5bZqqJqKbHyaBjYdzuemtLWxr4XIC36WcpmdXX0b07trqMrgTwwUc4HWgAggDFgJLRGRYI8cKcDsQBFwOPCAiN9c7Zq5SKsC2zXFUoduzzLNlnCosZ7ybdnvVHC8uOpQKs4UtR42/fstnO0/WjKKpMrdtzZmtx/Jr/l9lUdz/0U72nDzbrHPLKsxsOHyGOUPDOswPOUMFHBHxB64H/qCUKlZKbQK+BG5r6Hil1ItKqZ1KqSql1EHgC2Cy80rcMSSm29pvdA81rRHj+wXh5+3hFu04xeXWuzAPO6w5E9s/GG9PkzUvD+sg2J+88SMvrk6lvNLc5LkbDp+hvNLCnGEdo/0GQIzUlVFExgCblVJ+tdIeAaYrpeZe5FwBdgJvKaXetKWlA52wBtZdwKNKqeRGzl8ELAIICQmJWb58edtfUBOOFJhJzTczuJsHA4M8HPpcrVVcXExAQADvp5znx1NVvH6pHx6mjvFLrLbq69CRNecavJZUTmaxhRendTLsL3alFI9uKCPQRxgd4tHiz19D16H2Z7lngIlPUivYmFlFL3/hskgvzlWoBp/n7T3n2ZVTxd9n+uHpRp+rpt4LM2bMSFJKjWvsXKNNbRMAFNVLKwQ6N+PcxVgDy79qpS3EGoQEeAhYIyKDlVIX3PMqpZYCSwGio6NVXFxcS8veoA0Hz/DNvlP07NqJAB9Pcs6dJzWriE1HcrEo8PY08/E9xpz0MCEhgbi4OF7YvYHx/X24dOZEVxfJJaqvQ0fWnGtwwjeDP3y+j4jh4+kfYswAfSCriNw1G3n48uEsmNi3xec3dB3i6h1z1WxIOJjDb5Yn86+UCgA8pIq5o3oxIrwrIZ196B7gTXJ8IpEhXQgaMNyQn//GtOXz4NSAIyIJwPRGdm8Gfgl0qZfeBTh3kXwfwNqWM1Updb46XSm1udZhz4vIHcBU4KuWlbx1/rsni/s+2lknzcfThK+XqWbd8ooqC//Zcdywb7jCskoOZp/jyhE9XV0UzeDiokIAa/doowacdbZBlrMcPGdZXHQot8b25e/fH0EBZqX4MjmTz+sNFD2QVcTC/3PMcgdG5NQ2HKVUnFJKGtmmAIcATxEZVOu0UUCjc0iIyF3AE8ClSqmLrQalsN7tOFx+SQVPfr635rFJ4MGZA0n90+W8e+cEfL2s9b4CrNyZybd7s5xRrBbbebwApWCcHvCpXUSfbn70D/EnwcDdo9ceyGZ0n0BCuzh+zrJpUaH42D7nvl4mVvx8Erufms3aX0/jpnHh1St91My03REYqtOAUqoEWAk8IyL+IjIZuBb4oKHjRWQh8GdgtlIqrd6+viIyWUS8RcTX1mW6O9Y7KYcqqzBz13s7KD5fhbeH9Q3n7WlienQoIkJMRBDL7o7l4TnR/PunExjVJ5D7P9rJx9uPO7poLZaYno+HSRjdJ9DVRdHcQFxUKFvT8iiraLrB3BVOF5az52Qhs4c6Z8xL7c/5srtjiYnsRqCfN4PCOjN/fN+aYNSR1pYyWhsOwH3Au0AOkAfcq5RKARCRqcC3Sqnq+/VngWBgR61Gyg+VUr/A2u6zBBgAlAO7gSuUUg79KVFltvDLj3eRfPIsSxbGENLZh61pecT2D65zyxwTEVTzeFy/IO5btpPfrtxLQWkF904fYJhG1x3pBQzv1QU/byO+VTSjiYsO4d3Nx9h6LI8ZBlu5cu0Ba3XaHCcFHKj7Oa+fvuzu2Aa/G9ozw32LKKXygXmN7NuItWNB9eN+TeSTAoy0ewGboJTiD1+ksO5ANs9cO6xmBPPF3kx+3p68ffs4HlmRzIurD5KaVUR0j87E9u/u0jdilUWRfOIst8ZGuKwMmnuZ0K8bvl4mfjh4xngBZ382EcF+DAw1RvtSY8GoPTNcwHFn/1x/hI+3H+e+uAHcPimyRed6eZj46/zRVFRZ+DI5C0nOwsfriEsbE9OLLJyvsugJO7Vm8/XyYFL/YBIO5gCNjdd2vnPllWw5mssdkyINU3vQERmqDcddJWUUcO+HSbyy9hDXjenNo5dFtyofk0kY3tvaSc8IjYmHC6zru8foCTu1FoiLDiU9r5T03BJXF6XGhkO5VJqV09pvtIbpO5w2Ssoo4JalW6kwWzAJ3DS+T5t+QcX2746P5xHOV1kcsnZ6SxwqMNOvuz8hnVs2k67WscVFW7tH/3DoDJEGmXJ/7f7TBPl5dbgqLKPRdzhtFJ+aQ4XZeicgQGIbJgIEa73uR/fEEtHNj8BOXi7rHaaU4nCBWX9AtRaLCPanZ1df/v1jepsmxrSXSrOF9ak5zBwchqeH/spzJX3120ApxY50a5WXyY7dG2MigvjtlYPJLalgfWpOm/NrjaNnSiiuRLffaC2WlFFAzrnzpOWWsPDtrS4POjuO5VNUXsXsocbqxNAR6YDTgMLzqlkfkuWJJ9h2rIDbJ0Xwm+q+9na6I5g1JIywLj58uDXDLvm1VFKGnrBTa52taXk1y02fN8CgxrUHsvH2NDF1UIhLy6HpgNOggvOKBRf5ZZZ2ppjFX+7nkgHBLJ47jPtnDLRr9ZOnh4mbx/dlw+EzHM9r2xK2rbF632m8TXC2pMLpz625t+oZlKu58i5ZKcXa/dlMGdgdfx/dZO1qOuA04nyVhc93ZTa4r9Js4Vf/2Y23p4lX54/G5KCZXm+Z0BeTCMu2O/cuJyk9n4SDZ6iwwMJ3trm8SkRzL9WDGueN7oUCcotd96Ml9fQ5ThaU6d5pBqEDTiME+HjHcZbvOHHBvtfWHWLPyUJeuG4EPbo6bk6mHl19mTUklBWJJzlf5bypQr7ak1WzQJWru2Zr7ikmIohX5o8mItiPtzemXfwEB1lrm6zzUgdP1qk1jw44DQjyEd796Xhi+wXz2Gd7eHLVXiqqrD3RtqXl8UbCUW4a14crnDCD8q2xEeSXVPDt3tMOf65q1csEm+hY8zxp9uVhEn42pR+7jp+taRN0tnXVk3V2dvxkndrF6YDTgK4+wozoUN776Xh+Pr0/y7Yd55a3t/Ll7kzueT+RsM4+PDV3qFPKMnlAdyKD/ZzaeWDPyUKG9OzMdYO8Osy06Zpj3BATTtdOXvzfxmNOf+61KdnsOVnI0F71VzzRXEUHnCZ4epj47RVDeH3BWPZlFvLgJ7spKq8iv7SS1NNNLtFjNyaTsHBiBIkZBaSerr82nf0dyy0h9fQ5bozpw9UDvHWw0drEz9uThRP7sibltFM7vyRlFHDvsiQAPks6qdshDUIHnGa4amRPFkz43+qAZrNz2zVuiAnH29PklLucNSnWqrvLhnecddY1x7rjkkg8TMK7m513l7M1LY8q2yqHVU7+vGqN0wGnma4e1atm0TRnt2sE+Xtz9cierNqZSfH5Koc+1+p9pxkZ3pXegZ0c+jxaxxHWxZe5o3qxPPEEhaWVTnnO6hk6BN0OaSQ64DTTBYspObmq6dbYCEoqzI121baHrMIydp84y2XD9N2NZl93T+lPaYWZj5y0yGD1D7P548J1O6SB6JFQLeDK9SvG9AlkaM8uvL0hjcKyCoeslbNmn7U67QpdnabZ2dBeXZg8MJj3fjzGz6b0qzMw1BHWH8ihs48nf5o3wuHPpTWf/ku4CRFh2qDuZOSX8sp3h1j4f/afo2p1ymmiwgLoH2KMBaq09uXuqf3JLjrPN3tPOfR5LBbF+oM5TIsO0cHGYAz31xCRbiKySkRKRCRDRBY0cexiEakUkeJaW/9a+0eLSJKIlNr+He2cV+EYvt4eAFiU/Qdk5hWfZ/uxfC7X1Wmag0wfFMLA0AD+tu4wr8cfdljPsb2ZhZw5d55LB+vBnkZjuIADvA5UAGHAQmCJiDS1dOB/lFIBtbY0ABHxBr4APgSCgH8DX9jS3dLUQSF4eVin0TGZ7LtWzroD2ViU7p2mOY7JJMweYl2czVF36QDfp+ZgEutCcJqxGCrgiIg/cD3wB6VUsVJqE/AlcFsrsovD2kb1mlLqvFLq71g7rcy0V3mdzdpxYSLd/L3pHuDDyPCudst79b7T9OnWiaE99SA5zXF8vRx3l15tfWo2Y/sG0c3fbX9btltG6zQQBVQppQ7VSksGpjdxzlwRyQeygH8qpZbY0ocBe1T1POlWe2zpq+tnIiKLgEUAISEhJCQktPpFONrt0cJrO8v544ffMyvCq835lVYqNhwqZXaEJz/88ENNenFxsaGvg7Po62C/a+B/zoxJrAHHJOBzNoOEhJNtL6BNQbmFfZll3BDl5ZC/mX4vtO0aGC3gBAD1h9MXAp0bOX45sBTIBiYCn4nIWaXUx7a8Cpubl1JqqS0voqOjVVxcXGvK7xTTlWJ74Ta+ySjisZsm08W3bUHni92ZmNVu7rliQp2ebwkJCRj5OjiLvg72uwZxQGi/TB78ZDc3T4jg7nnD25xnbR9tOw7sZdFVk4gKa+xro/X0e6Ft18CpVWoikiAiqpFtE1AM1K/T6QI0OI+MUmq/UuqUUsqslPoR+Btwg213i/JyJyLC764cwtmySt6IP9rm/FbvO01oZx/GuGg5a61juWZ0b8ZHBrH5SC51KyDa7vsD2YQHdWJQqO5paURODThKqTillDSyTQEOAZ4iMqjWaaOAlOY+BdZ2GmznjBSR2ovVjGxBXoY2vHdXfjKmN+9uPsbJgtbPUVVWYSbh4BkuG9bDYev6aFp988f1IS23hB3p9us0UFZhZtORXGYNCaPux14zCkN1GlBKlQArgWdExF9EJgPXAh80dLyIXCsiQWI1AXgQa880gATADDwoIj4i8oAtfb1DX4QTPTInGgFeXnOw1XlsOHyGskozl+veaZoTXTWyJwE+nnyyw34zD2xJy+V8lYWZuju0YRkq4NjcB3QCcoCPgXuVUikAIjJVRIprHXszcARrNdn7wF+UUv8GUEpVAPOA24GzwF3APFt6u9ArsBN3T+3H57tPsefk2VblsXrfaQL9vJjQr5udS6dpjfPz9uSa0b34794sisrtM7/augM5+Ht7MLG/fi8bleECjlIqXyk1Tynlr5Tqq5T6qNa+jUqpgFqPb1FKBdvG3wy2dX2undcupVSMUqqTUmqsUmqXM1+LM/xi+gCC/b157psDLa4P35aWxzd7sxjbJxAvD8O9FbR27qZxfSivtPDl7rbPPKCUYv2BHKYOCsHH08MOpdMcQX/LuLnOvl78anYU247l8/Dy5GYPpEvKKODWd7ZRUWVh45FcvV6I5nQjw7syuEdn/tPAMu4tlXKqiNNF5czUS0kbmg447UB0WAACrNqVyU1vbeHHI7lNHr//VBG/Wb6bSrP1jshiUXq9EM3pRISbxvdhb2YhKafqj2BomfWpOYjADD27gKHpgNMO7EgvoLpTTpVF8dP3dvDi6lRyisrrHJdVWMYjK5K56h8bOXPuPJ4mccn6PppW7SdjeuPtaWJ5G+9yvk/NYVR4ICGdfexUMs0RjDbwU2uF2P7BeHuaqKyy4OFhYnSfQJb8cJS3N6ZxzajejIsM4ovdmezMKACEe6b25/64gRw5U8zWtDxi+wfr9UI0lwj08+byYT1YtSuT3145pGbqm5bIOVdO8omz/GZ2lANKqNmTDjjtQPXicLWDR0ZeCe9uOsbH20/w2U7r1CEmgdcXjOGKET1rztOBRnO1m8b34cvkU6xJOc21o3u3+PyE1DMAXDokzN5F0+xMB5x2on7wiAj254/XDqdLJy/+uf5IzYjYtNwSl5VR0xoyqX8wfbp14pPtJ1oVcD5NOkFnH0/KKhy7/LrWdroNp52Liw7Fx8uk22o0wzKZhPkxfdiSlkdGHxOW/AAACrlJREFUXst+EG05msv29ALOna9i4TvbdG9Lg9MBp52rrm57eE60XttdM6wbxoUjwOOf7WlR0Hj5u/9NLO+o5Q40+9EBpwOIiQji/hkDdbDRDOvU2XJEYGtafrMXZvto23GSMgrw0L0t3YZuw9E0zeW2puVRPVFGeaWFjYfPNPkD6cejuTz1xT6mR4Vw/4wB7Egv0L0t3YAOOJqmuVxs/2B8vEycr7SggK/3ZHFbbATBAReOqzmWW8K9H+4ksrs//1gwhi6+Xkzop+9s3IGuUtM0zeWq2xofuSyaxy+P5kR+KTe8uYXjeXWX3igsq+Rn/96BSeCdO8a1efFBzbl0wNE0zRCq2xrvjRvIR/dMpKC0guuWbGbvSeu0N1VmCw98tJMT+aW8eWsMEcH+Li6x1lI64GiaZjgxEd349BeX4OPpwc1Lt/DOpjTmv7WFjYdzeW7eCCbqzgFuSQccTdMMaWBoACvvu4TuAT786esD7Dx+Fg+TMEAvH+22dMDRNM2wwrr4cu2YXv9LUHpmc3emA46maYY2PSoUXz1bRrtguIAjIt1EZJWIlIhIhogsaOLYb0WkuNZWISJ7a+1PF5GyWvu/c86r0DTNXvRsGe2HEcfhvA5UAGHAaOAbEUlWSqXUP1ApdUXtxyKSAKyvd9hcpdQ6B5VV0zQn0DObtw+GusMREX/geuAPSqlipdQm4EvgtmacGwlMBd53ZBk1TdO01jFUwAGigCql1KFaacnAsGacezuwUSmVXi99mYicEZHvRGSUncqpaZqmtZDRqtQCgKJ6aYVA52acezvwbL20hcBOrEvBPASsEZHBSqmz9U8WkUXAIoCQkBASEhJaVvJ2qLi4WF8H9HUAfQ2q6evQxmuglHLaBiQAqpFtEzAGKK13zm+Ary6S7xSgGAi4yHGpWNt0mixnVFSU0pSKj493dREMQV8HfQ2q6evQ9DUAElUT361OvcNRSsU1td/WhuMpIoOUUodtyaOACzoM1HMHsFIpVXyxImC929E0TdOczFBtOEqpEmAl8IyI+IvIZOBa4IPGzhGRTsB84L166X1FZLKIeIuIr4g8CnQHNjvsBWiapmmNMlTAsbkP6ATkAB8D9ypbl2gRmSoi9e9i5gFngfh66Z2BJUABkAlcDlyhlNLDlDVN01zAaJ0GUErlYw0i/9/e/YbIddVhHP8+dmui2caQtEZoMUtq05otqYkVQQ0GGgn2haB5UzZKA9Xgn4DYNy2oZKuhRRB8Ia0a3NKaSKu0MSIUXwhGTCHSYFPrtjElQuOfKDQ0yXbTrH/4+eKcMeO4u83ezZw7s/f5wIW990yG330y3N/M3bNnphv7NWliQfuxx0iNqfOx48C6btRoZmZz14ufcMzMbAFywzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Mi3HDMzKwINxwzMyvCDcfMzIpwwzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzInqu4UjaKemIpClJj1zC478k6W+Szkl6WNKitrEhSb+UdF7SMUmbu1q8mZnNqOcaDvBXYDfw8Bs9UNIW4F7gNmAVsBq4r+0hjwHPAiuALwNPSLrmchdsZmZvrOcaTkTsj4gDwOlLePidwFhEjEfEq8DXge0AktYAG4BdEfF6RDwJPA9s7U7lZmY2m4G6C5inYeCnbfvPASslrchjf4yIiY7x4emeSNIOYEfenZL0+y7U22+uBl6pu4ge4BycQYtzmD2DVbP9w35vOIPA2bb91s9XTTPWGr92uieKiD3AHgBJRyLi1stbav9xDolzcAYtzmF+GRS9pSbpoKSYYTtU4SlfA5a27bd+nphmrDU+gZmZFVe04UTEpojQDNuHKjzlOHBL2/4twN8j4nQeWy3pqo7x8epnYGZmVfXcpAFJA5IWA1cAV0haLGmmW38/AO6StFbSMuArwCMAEXEcOArsys/xcWAd8OQllLFnvuexQDiHxDk4gxbnMI8MFBGXs5B5kzQK7Oo4fF9EjEp6J/ACsDYiTubH3w3cA7yF1Ew+GxFTeWyI1IDeD5wEvhARv+j+WZiZWaeeazhmZrYw9dwtNTMzW5jccMzMrAg3nDaSlkv6iaRJSS9LGqm7phJmW79O0m15HbrzeV26Wf+wq19JWiRpLP+/T0g6KumjbeNNyWGfpFN5bcLjkj7dNtaIDNpJukHSBUn72o6N5NfJpKQDkpbXWWM35T9luSDptbz9oW1szjm44fyvB4F/ACuBbcB3JE27MsECM+36dZKuBvYDXwWWA0eAHxWvrowB4E/Ah4G3kWY8/jgvANukHB4AhiJiKfAxYLek9zYsg3YPAs+0dvL14HvAp0jXifPAQ/WUVszOiBjM241QPQdPGsgkLQFeBW7OU6qRtBf4S0TcW2txhUjaDVwXEdvz/g5ge0R8IO8vIS1psT4ijtVWaCGSfkdaDHYFDcxB0o3AQeCLwDIaloGkO4BPkGbGvisiPinpflJDHsmPuR54EVjRsYzWgiDpILAvIr7fcbxSDv6Ec9Ea4F+tZpPNuPZaQwyTMgAgIiaBEzQgE0krSa+JcRqWg6SHJJ0HjgGngKdoXgZLga8Bd3cMdeZwgnRXZE256op7QNIrkp6WtCkfq5SDG85Fg8C5jmNnSeuyNdVM69Et6EwkXQn8EHg0v3tvVA4R8XnSuW0k3UabomEZkFaeH4uIP3ccb1oO95C+9uVa0h98/ix/mqmUgxvORV577f81LhNJbwL2kt6t7cyHG5dDRPw7Ig4B1wGfo0EZSHoPsBn41jTDjckBICJ+ExETETEVEY8CTwO3UzGHfl8t+nI6DgxIuiEiXsrHmr722jjpO4eA/963v54FmokkAWOkX4LeHhH/zEONyqHDABfPtSkZbAKGgJPpJcEgaZmttcDPaVu/UdJqYBHp+tEEAYiOdSwvOYeI8JY34HHSt4QuAT5I+og4XHddBc57AFhMmqG0N/88AFyTM9iaj30DOFx3vV3M4bvAYWCw43gjcgDeDtxBvsACW4BJ0my1RmSQc3gr8I627ZvAEzmDYdKt9435OrEPeLzumruUw7L8GmhdD7bl18OaqjnUflK9tJGmex7IoZ4ERuquqdB5j+Z3Lu3baB7bTPrl8eukGUtDddfbpQxW5fO+QLpd0Nq2NSWHfEH9FXAmX0yeBz7TNr7gM5ghl1HSTK3W/ki+PkySvgByed01dvH18AzpNtmZ/GbsI/PJwdOizcysCE8aMDOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Mi3HDMzKwINxwzMyvCDcesT0haKmlU0rvrrsWsCjccs/5xK7ALuLLuQsyqcMMx6x/rSV8V8ELdhZhV4aVtzPqApBeBmzoO74+IrXXUY1aFG45ZH5D0PtJq5uPA/fnwqYh4ub6qzObG34dj1h+eI30Z2rcj4nDdxZhV4d/hmPWHYeDNwG/rLsSsKjccs/6wgfR9PUfrLsSsKjccs/6wHjgREefqLsSsKjccs/6wFk+Htj7nSQNm/eEMsEHSFuAs8FJEnK65JrM58bRosz4g6WZgDFgHLAY2RsSheqsymxs3HDMzK8K/wzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrwg3HzMyK+A/Wyj+eeqsl9QAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "9Zi14gly5jk6" }, "source": [ "Another simple approach is to use a fully connected network. Since it expects a flat list of features for each input, we need to add a `Flatten layer`. Let’s just use a simple Linear Regression model so that each prediction will be a linear combination of the values in the time series:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "Tr1yTxkj5jk6", "outputId": "7368b75f-6b0c-4019-9113-96586a7c23f2", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 3s 5ms/step - loss: 0.1001 - val_loss: 0.0545\n", "Epoch 2/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0379 - val_loss: 0.0266\n", "Epoch 3/20\n", "219/219 [==============================] - 1s 4ms/step - loss: 0.0202 - val_loss: 0.0157\n", "Epoch 4/20\n", "219/219 [==============================] - 1s 4ms/step - loss: 0.0131 - val_loss: 0.0116\n", "Epoch 5/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0103 - val_loss: 0.0098\n", "Epoch 6/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0089 - val_loss: 0.0087\n", "Epoch 7/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0080 - val_loss: 0.0079\n", "Epoch 8/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0073 - val_loss: 0.0071\n", "Epoch 9/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0066 - val_loss: 0.0066\n", "Epoch 10/20\n", "219/219 [==============================] - 1s 4ms/step - loss: 0.0061 - val_loss: 0.0062\n", "Epoch 11/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0057 - val_loss: 0.0057\n", "Epoch 12/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0054 - val_loss: 0.0055\n", "Epoch 13/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0052 - val_loss: 0.0052\n", "Epoch 14/20\n", "219/219 [==============================] - 1s 4ms/step - loss: 0.0049 - val_loss: 0.0049\n", "Epoch 15/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0048 - val_loss: 0.0048\n", "Epoch 16/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0046 - val_loss: 0.0048\n", "Epoch 17/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0045 - val_loss: 0.0045\n", "Epoch 18/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0044 - val_loss: 0.0044\n", "Epoch 19/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0043 - val_loss: 0.0043\n", "Epoch 20/20\n", "219/219 [==============================] - 1s 3ms/step - loss: 0.0042 - val_loss: 0.0042\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.Flatten(input_shape=[50, 1]),\n", " keras.layers.Dense(1)\n", "])\n", "\n", "model.compile(loss=\"mse\", optimizer=\"adam\")\n", "history = model.fit(X_train, y_train, epochs=20,\n", " validation_data=(X_valid, y_valid))" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "swohpJKX5jk6", "outputId": "df2883b3-43eb-432e-a2cf-5188ae484c6a", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 0s 2ms/step - loss: 0.0042\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "0.004168086219578981" ] }, "metadata": {}, "execution_count": 10 } ], "source": [ "model.evaluate(X_valid, y_valid)" ] }, { "cell_type": "markdown", "source": [ "If we compile this model using the MSE loss and the default Adam optimizer, then fit it on the training set for 20 epochs and evaluate it on the validation set, we get an MSE of about 0.004. That’s much better than the naive approach!" ], "metadata": { "id": "nDRQiwp7_7Zd" } }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "kckEzMwk5jk7", "outputId": "cbed5fa3-36cc-4f07-dc94-eb94a923594e", "colab": { "base_uri": "https://localhost:8080/", "height": 291 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUVf7/8ddn0klIIxBqqANISWiKQZAgxYoNdBVUUBTLT1dWEXW/a0NXFwVddVFQQVAQ0BVWVFQ0EqVEBBEUVJqA1NAhCSmTzPn9cYcwiQOkTYF8no/HfWTm3jP3vicZ5sO999xzxRiDUkopVd1s/g6glFLq7KQFRimllFdogVFKKeUVWmCUUkp5hRYYpZRSXqEFRimllFdogVFKKeUVPi0wIhIvIvNEJFdEtonIkJO0ExEZJyIHXNM4ERG35ca1jhzX9Jbv3oVSSqnyCPbx9iYChUAi0An4VETWGGPWlWk3ErgaSAEM8CWwBZjk1ibFGLPJ+5GVUkpVhs/2YEQkEhgEPGaMyTHGLAHmAzd7aD4MmGCM2WGM2QlMAIb7KqtSSqmq8+UeTGugyBizwW3eGqC3h7btXcvc27Uv0+ZbEbEBy4AHjDFbPW1UREZi7RERERHRtXlsLBG7d5PbtClbs+Kw2QyNG+eV+004nU5stsA4dRVIWSCw8mgWzwIpCwRWHs3i2YYNG/YbY+pW6sXGGJ9MQC9gT5l5dwAZHtoWA23dntuxDpWJ6/mFQCgQC/wHWAsEny5D165djfnhB2PAmP/+1wwYYEz37qZCFi1aVLEXeFEgZTEmsPJoFs8CKYsxgZVHs3gGrDSV/N73ZYnMAaLLzIsGssvRNhrIcb1ZjDHfGmMKjTGHgfuB5sA55Upht1s/N24kLg4OHarAO1BKKVVuviwwG4BgEbG7zUsByp7gxzUvpRztjjOAnGL5CbVrQ/36sHEjsbFw+HC5XqWUUqqCfFZgjDG5wFxgrIhEisgFwFXAux6avwM8ICKNRKQh8CAwDUBE2otIJxEJEpEorA4AO4Ffyx3GbocNG0r2YPSOBUopVf18fRbpHiAC2AvMAu42xqwTkV4ikuPWbjLwMfAz1vmVT13zwOriPAc4CvwONAOuMMY4yp3Cbi85ROZwwLFjVXtTSiml/syn18EYYw5iXd9Sdv5iIMrtuQHGuKaybb8G2lQpiN0OWVnUDTsKRHP4MERGVmmNSimlyvD1hZaBoXVrABrnbwK6cOgQNGrk30hKlcfRo0fZu3cvDkf5d9jdxcTE8Ouv5T+a7G2BlKemZomMjKRx48Ze6RZdMwuMqydZ/aMbOF5glAp0R48eJSsri0aNGhEREYHb6Enllp2dTe3atb2QrnICKU9NzOJ0Otm5cyf79++nXr161b7+wLiSx9datgQg/sBGQHuSqTPD3r17adSoEbVq1apUcVGqLJvNRmJiIkeOHPHO+r2y1kBXqxY0bkx0llVgdA9GnQkcDgcRERH+jqHOMiEhIRQVFXll3TWzwAC0bk3EDi0w6syiey6qunnzM1VzC4zdTtDv1rBoeohMKaWqX40uMHLwIE1rH9Q9GKWU8oKaW2BcXZU7RW7UAqPUGWb48OFcccUVFXpNWloa9957r5cSnfDkk0/SoUMHr2/nTFAzuylDSVfl9iEbWHu4u5/DKHV2Ot3x/SFDhjBz5swKr/fll18+PtJ6uc2dO5eQkJAKb0tVXs0tMC1agM1GG9tGFusejFJesXv37pLHn3zyCXfccUepeWV7LzkcjnIVgZiYmApniY+Pr/BrVNXU3ENkoaHQtCnNi/UQmapZMjPhueesn95Wv379kik2NrbUvPz8fJo0acKsWbO46KKLiIiIYPLkyRw4cIAbb7yRxo0bExERQfv27Xn77bdLrbfsIbK0tDTuuece/v73v5OQkEC9evUYPXo0TqezVBv3Q2TNmjXjmWee4c477yQ6Opq2bdvywgsvlNrOhg0b6N27N+Hh4bRp04YFCxYQFRXFtGnTyv07cDqdPP300zRp0oSwsDA6duzIRx99VKrN2LFjadq0KWFhYdSvX5+RI0eWLPv22285//zziYqKIiYmhvPOO4+1a9eWe/v+VHP3YABat6bJyo1oJzJ1Jho1ClavrthrDh2KYO1acDrBZoPkZKjIzkCnTvDvf1dsm6fz6KOPMn78eKZMmUJISAj5+fl06dKFhx9+mOjoaL766ivuvPNOkpKS6Nu370nXM3PmTO6//36WLVvG6tWrGTJkCF27duXGG2886WteeuklnnrqKR566CHmzZvHmDFj6NmzJ6mpqTidTq655hrq16/Pd999R15eHqNGjaKgoKBC7+/ll1/mhRdeYNKkSXTr1o0ZM2Zw7bXX8sMPP9CpUyc+/PBDxo8fz6xZs+jYsSN79+4lIyMDsPbwrrrqKkaMGMHMmTNxOBysWrWKoKCgCmXwl5pdYOx26i9axqG88t9ORqkz2ZEjwvH/1DudcORIxQqMN9x3330MHjy41LyHHnqo5PHIkSP5+uuvmTVr1ikLTLt27Rg7diwArVu35s033yQ9Pf2UBWbAgAElezV33XUXb7zxBunp6aSmpvLll1+yfv16Fi5cSCPXYIUvvfQSF1xwQYXe3/jx4xk9ejRDhgwBrL2Vb7/9lvHjxzNjxgy2bdtGgwYNGDBgACEhISQlJdGmjTWe79GjRzl8+DADBw6kpWsEkrZt21Zo+/5U4wtMeGE2kYV7cTgS0fN/6kxSmT2Jr77K58orIykstI4Sz5wJqanVn60iunXrVup5cXEx//rXv5gzZw47d+6koKCAwsJC0tLSTrme5OTkUs8bNmzI3r17K/2a3377jYYNG5YUF4Bzzz23QoNCHj16lF27dv2pKPXs2ZMFCxYAcN111/Hyyy/TvHlzLr74Yi655BL69OlD7dq1iY+PZ/jw4Vx88cX07duXvn37MnjwYJKSksqdwZ9q7jkYKOlJZmejXmypaoTu3Z2kp8PTT0N6uv+LC1ij+bobP348EyZM4KGHHiI9PZ3Vq1dz9dVXU1hYeMr1lO0cICKlzsFU12uqy/Eedk2aNGH9+vVMnjyZ6OhoHnzwQS688EJyc3MBePvtt1m+fDkXXngh8+fPp02bNnzxxRc+yVhVNbvAuK6FsaMn+lXNkZoKjz4aGMXFkyVLljBw4EBuvvlmOnXqRMuWLdmwYYPPc7Rt25Zdu3axa9euknkrV66sUAGKjo6mYcOGLF26tNT8JUuW0K5du5Ln4eHhXH755bz00kusWLGCX3/9tdRrUlJSePjhh8nIyCAtLY3p06dX4Z35Ts0+RNa0Kc6gYFoXb9ACo1SAaN26NXPmzGHJkiUkJCTw6quvsmXLFjp37uzTHP3796dNmzYMGzaM8ePHk5eXxwMPPEBwcHCFxu966KGHePzxx7Hb7XTt2pUZM2awePFiVq1aBcC0adMoKiqie/fuREVFMWfOHEJCQrDb7WzZsoXJkydz5ZVX0qhRI37//Xd++ukn7r77bm+97WpVswtMcDAFDVtg366HyJQKFP/4xz/YsmULl156KREREQwfPpyhQ4fyyy+/+DSHzWZj3rx53H777Zx33nk0a9aMCRMmcO211xIeHl7u9fz1r38lOzubMWPGkJWVRZs2bfjwww9JSUkBIDY2lnHjxjF69GgcDgft2rVjxowZNG/enKysLDZs2MB1113H/v37SUxMZOjQoTz88MPeetvVyxhTY6auXbuaso6mXWFWk2xmzfrTIo8WLVpUvoY+EEhZjAmsPGdjll9++aXK6zh69Gg1JKk+gZSnPFlWr15tALNy5Uq/Z6lOp/psAStNJb9za/YeDGBrbceekc6yA05q+ikppVRp8+bNIzIyErvdztatW3nggQdISUmhS5cu/o52RqjxBSa0vZ0Q8ijevgto7O84SqkAkp2dzcMPP8z27duJi4sjLS2Nl156Se/LU041vsCEtLO6Kods3YgWGKWUu1tuuYVbbrnF3zHOWHpMyNVVOWLnRj8HUUqps4sWmMaNyZdwYrN8389eKaXOZlpgbDZ2hbck4ZDuwSilVHXSAgNkRdupn6MFRimlqpMWGGB/ndY0yt8MxcX+jqKUUmcNLTBAdqKdMArhjz/8HUUppc4aWmCA/CZWV2Xnej1MplQgevLJJ+nQocNJn3ty7733nnaI/8ps21vK3qXzbKAFBnA0t7oqF6zVAqNUdbryyitPepOwX3/9lejoaBYuXFjh9Y4ePZpvvvmmqvFK2bZtGyLCypUrvb6tmkILDBCaVJ9sonCs067KSlWnESNGsGjRIrZu3fqnZVOmTCEpKYl+/fpVeL1RUVHUqVOnGhIG1rbONlpggLh4YROtMBt1D0bVAJmZ8Nxz1k8vu/zyy0lMTOTtt98uNd/hcPDuu+9y0003YYxhxIgRNG/enIiICOx2O88///wp77tS9rBVcXExo0ePJi4ujri4OEaNGkVxmU47n3/+Ob169SIuLo74+Hguvvhifv3115LlHTt2BKy7VopIyeG1sttyOp08/fTTNGnShLCwMDp27MhHH31Usnzr1q2ICB9++CH9+/enVq1atGvXji+//LJCv7uCggJGjRpFYmIi4eHhnH/++SxZsqTU7/Cvf/0rDRs2JCwsjCZNmvDII4+ULJ87dy7JyclEREQQHx9P7969ycrKqlCGqqrxQ8UAxMbCRuy02fqjv6MoVX6jRsHq1RV6ScShQ7B2LTidYLNBcjLExJR/BZ06VehezcHBwQwbNoxp06bxxBNPlNxu+OOPP2b//v3cdNNNOJ1OGjVqxPvvv0/dunX5/vvvGTlyJHXq1GHEiBHl2s6ECRN48803efPNN0lOTmbixInMnDmz1KCUubm5jBo1iuTkZPLy8njmmWcYOHAgv/zyC6GhoSxatIg+ffrw+eefk5KSQmhoqMdtvfzyy7zwwgtMmjSJbt26MWPGDK699lp++OEHOnXqVNLu//7v/3jhhRd47bXXeOaZZ7jhhhvYtm0bUVFR5XpPY8aM4f3332fq1Km0aNGCF198kUsuuYSNGzfSoEEDXnnlFebNm8fs2bNp1qwZO3bsYP369QDs2bOHG264geeee45BgwaRk5PDd999V67tVifdgwHi4mADrQnfvQUcDn/HUcpr5MgRq7iA9fPIEa9vc8SIEfzxxx989dVXJfOmTJnCgAEDaNy4MSEhIYwdO5Zzzz2XZs2acf3113PXXXcxa9ascm/j3//+N2PGjOH666+nbdu2vPzyy9SvX79Um0GDBjFo0CDsdjvJycm8/fbbbNmyhe+//x6AhIQEAOrUqUP9+vWJj4/3uK3x48czevRohgwZQuvWrRk7diy9evVi/Pjxpdr97W9/Y+DAgdjtdp599lkOHjzI6nL+hyA3N5fXX3+dcePGcfnll3POOecwadIkEhMTmThxImCdM2rdujW9evUiKSmJHj16cOuttwKwa9cuHA4HgwcPplmzZnTo0IHbb7+dxMTEcv9Oq4NP92BEJB6YAgwA9gOPGmPe89BOgH8Bt7tmvQU84ro3gXu7W4DpwB3GmLcqmysuztqDsTmLYcuWkvHJlApoFdiTOC7/q6+IvPJKKCyE0FCYOdPr90622+307t2bqVOnMmDAAHbt2sUXX3zB7NmzS9pMmjSJt956i23btpGXl4fD4aBp06blWv+RI0fYvXs3qW7vw2az0b17d7Zv314yb/PmzTz22GMsX76cffv24XQ6cTqd/FGByxOOHj3Krl27uOCCC0rN79mzJwsWLCg1Lzk5ueRxw4YNAdi7d2+5trNlyxYcDkep7QQFBZGamlpy47Xhw4fTv39/WrduzYABA7jsssu49NJLsdlspKSk0K9fPzp06MCAAQPo168fgwcPpm7duuV+r9XB13swE4FCIBEYCrwuIu09tBsJXA2kAMnAQOBO9wYiEgf8HVhX1VDHD5EBoOdh1FnM2b07pKfD009bP71cXI4bMWIE//vf/zh48CDTpk0jPj6eq666CoA5c+YwatQohg8fzhdffMHq1au55557KCwsrNYMV1xxBfv27WPy5MksX76cH3/8keDg4GrbTtkh/ENCQv607FTnlSq6nS5durB161aee+45nE4nw4YNo3///jidToKCgli4cCELFy4kOTmZKVOmYLfbWbNmTZW3XxE+KzAiEgkMAh4zxuQYY5YA84GbPTQfBkwwxuwwxuwEJgDDy7R5DngFa0+oSmrXht9tWmBUDZGaCo8+6rPiAjB48GDCw8OZMWMGU6dO5ZZbbin5Al6yZAndu3fn3nvvpUuXLrRq1YrNmzeXe90xMTE0aNCg1DkGY0zJoS+AAwcO8Ntvv/H3v/+dfv36cc4555CdnU1RUVFJm+PnXMp2DnAXHR1Nw4YNWbp0aan5S5YsoV27duXOfDrNmzcnNDS01HaKi4vJzMwstZ3atWszePBgXn/9dT799FO+/vprNm3aBFiFKDU1lSeeeIIVK1bQsGFD5syZU20Zy8OXh8haA0XGGPe+wGuA3h7atnctc29XsqcjIucB3YB7gOtPtVERGYm1R0RiYiIZGRke2+VF9iAnP4YjGRlsdDtRV1ZOTs5J1+FrgZQFAivP2ZglJiaG7OzsKq2juLi4yuuorMGDB/PEE09w+PBhbrjhBrKzsykuLiYpKYlp06bx4Ycf0qJFCz788EO++eYbYmNjS7IWFBTgdDpP+vyuu+7i+eefp0mTJrRv354333yT3bt3U69ePbKzswkODqZOnTq89tprxMXFsXv3bv7xj38QHBxMfn4+2dnZxMfHExERwfz580lISCAsLIyYmJg/beu+++7j2WefpXHjxnTq1Ik5c+awePFiFi9eTHZ2Njk5OYB1HqXs7zovL++kv3+Hw0FRURHZ2dmEh4czYsQIxowZQ61atWjWrBkTJ04kKyuLW265hezsbP7zn/+QmJhIcnIywcHBTJs2jejoaGJiYkhPTycjI4O+fftSr149fvrpJ7Zv307z5s09bj8/P987/14qe6/lik5AL2BPmXl3ABke2hYDbd2e2wEDCBAErATOdy3LAG4vT4auXbue9L7TLVsasyn+XGP69TtpG2POznu9V5dAynM2ZjnVfdPLy9f3enf3ww8/GMD06NGjVJ6CggJz2223mdjYWBMTE2Nuu+0289RTT5mmTZuWtHviiSdM+/btT/rc4XCYUaNGmZiYGBMTE2Puvfdec9ddd5nevXuXtElPTzft27c3YWFhpn379ubzzz83kZGR5u233y7J8uabb5omTZoYm81W8tqy2youLjZjx441jRs3NiEhIaZDhw5m3rx5Jcu3bNliALNixYpS7x8wH3zwwUl/P8OGDTOXX355SZb8/Hxz//33m3r16pnQ0FDTvXt3s3jx4pL2b7zxhuncubOJiooytWvXNhdeeKFZunSpMcb6rFxyySUlr23ZsqUZN27cSbd9qs8WsNJU9nu/si+s8IagM3CszLwHgY89tD0CnOf2vCuQ7Xp8HzDVbVm1FJhu3Yz5usEQY9w+1J6cjV9c1SWQ8pyNWc70AuNJIOWpyVm8VWB8eZJ/AxAsIna3eSl4Pkm/zrXMU7u+wDUiskdE9gA9gAki8p+qhIuLg01Bra0BL/Pzq7IqpZRS+PAcjDEmV0TmAmNF5HagE3AVVoEo6x3gARFZgHVo7EHgVdey4UC4W9u5wH+xuj9XWmwsrC+2gzGweTO099S5TSmlVHn5upvyPUAEsBeYBdxtjFknIr1EJMet3WTgY+BnYC3wqWsexpjDxpg9xyesbs9HjTFVumIsLg7WFmhPMqWUqi4+vdDSGHMQ6/qWsvMXA1Fuzw0wxjWdbp1p1ZEtLg7mZWuBUUqp6qJDxbgcOQL7HLEUxtbVAqMClik9mIVSVebNz5QWGKxBZae4zuD8cMTOkZU6bL8KPCEhIeTl5fk7hjrLOBwOgoO9czBLCwyQkQHHL97dYOzYNusejAo89erVY+fOnRw7dkz3ZFS1cDqdZGVlEVOREbUrQIfrB9LSICwM8vJgk81O7aPTITcXIiP9HU2pEtHR0cCJkXIrIz8/n/Dw8NM39JFAylNTs0RGRpaMJF3dtMBgDcmUng7XXgtFUa1hE7BpE6SknPa1SvlSdHR0SaGpjIyMDDp37lyNiaomkPJoluqnh8hcUlPhmmtg8W5XT7INeh5GKaWqQguMmx49YHVuK+uJ9iRTSqkq0QLj5oILIJcocmMaaoFRSqkq0gLjplkzqF8f/giz6yEypZSqIi0wbkSsvZif8uy6B6OUUlWkBaaMHj3gh2w77NtnXd6vlFKqUrTAlHHBBbCB1tYT3YtRSqlK0wJTRufO8EeodlVWSqmq0gJTRmgoxJ/bEieiezBKKVUFWmA86NYznO00oeg3LTBKKVVZWmA8OH4eJm+NFhillKosLTAepKbCRuyEbNlg3UJZKaVUhWmB8SAhAY7UtROefxgOHPB3HKWUOiNpgTmJiBSrq7LZoIfJlFKqMrTAnESjNKurctZi7aqslFKVoQXmJDoMbE4RQWQt0T0YpZSqDC0wJ9GmQwh/2JpRuE4LjFJKVYYWmJOw2eBgndZE7tYCo5RSlaEF5hRMKztJ+Rs4eEC7KiulVEVpgTmF2HPtRJHLj5/t8XcUpZQ642iBOYUmF1k9yX7/Qg+TKaVURWmBOYXwZOtamMMrtMAopVRFaYE5laQkioJCsW3egMPh7zBKKXVm0QJzKkFBHKvfguZFG1mzxt9hlFLqzKIF5jRC2rfGzkaWLvV3EqWUOrNogTmNiI527LKJzKVOf0dRSqkzihaY07HbCTf5/P7tDh25XymlKkALzOnYra7KtbM2sn27n7MopdQZxKcFRkTiRWSeiOSKyDYRGXKSdiIi40TkgGsaJyLiWpYgIktd8w+LSKaIXOC10K2trsp6HkYppSrG13swE4FCIBEYCrwuIu09tBsJXA2kAMnAQOBO17Ic4DagLhAHjAM+FpFgryRu2BATEUGHkA0sW+aVLSil1FnJZwVGRCKBQcBjxpgcY8wSYD5ws4fmw4AJxpgdxpidwARgOIAxJt8Ys94Y4wQEKMYqNPFeCW6zIa1a0TVmoxYYpZSqADE+OnMtIp2BpcaYWm7zRgO9jTEDy7Q9Agwwxix3Pe8GLDLG1HZr8xPQFggB3jLG3HGS7Y7E2iMiMTGx6+zZsyucvf3jj+NYs5MmOZuYPftz6taNqPA6vCEnJ4eoqCh/xygRSHk0i2eBlAUCK49m8axPnz4/GGO6VerFxhifTEAvYE+ZeXcAGR7aFgNt3Z7bAYOrILrNDwduBIaVJ0PXrl1NpTzyiCkODjFBOMyECT9Wbh1esGjRIn9HKCWQ8mgWzwIpizGBlUezeAasNJX83vfOeQvPcoDoMvOigexytI0GclxvtoQxJh+YJSK/ishqY4x3rre327EVOWjGNtaujfHKJpRS6mzjy5P8G4BgEbG7zUsB1nlou8617HTtjgsBWlQ54cm4uir3b7aRdevK1killFKe+KzAGGNygbnAWBGJdHUtvgp410Pzd4AHRKSRiDQEHgSmAYjI+SLSU0RCRSRCRB7G6pW23GvhXV2VezfayLp1MTj1on6llDotX3dTvgeIAPYCs4C7jTHrRKSXiOS4tZsMfAz8DKwFPnXNAwjD6u58ANgJXAZcbozZ5bXU9epB7dp0qrWR3NxgfvnFa1tSSqmzhi/PwWCMOYh1fUvZ+YuBKLfnBhjjmsq2/YbSh8+8TwTs1u2TAZYuhQ4dfJpAKaXOODpUTHnZ7UTs3EhcXKFeD6OUUuWgBaa8WrdGtm6lU7t9OmSMUkqVgxaY8rLbwemkd5O1bN4MWVn+DqSUUoGtygVGREKqI0jAc3VV7hbzK4AeJlNKqdOoUIERkb+KyCC351OAPBFZLyJtqj1dIHEVmFZmI2FhWmCUUup0KroH81dgH4CIXAhcDwwBVmMNSHn2qlMH4uOJ2rODbt3Q8zBKKXUaFS0wjYAtrscDgQ+MMe8DTwLnV2OuwFS/PvErVnBjs0x++AHy8/0dSCmlAldFC8xRoJ7rcX8g3fXYgTXw5NkrMxPWryd8927u/KAvXQqtIqOUUsqzihaYhcCbIvIW0Ar4zDW/PSf2bM5OGRngdCJAUFEhaWToeRillDqFihaY/wcsxbqb5GDXlfkAXbCGfjl7paVBeLh1zwCbsLlxmp6HUUqpU6jQUDHGmKPAfR7mP1FtiQJVaiqkp5M7ZAhRhw8TfVE35n8GxlgjySillCqtot2U27l3RxaR/iIyQ0QeFZGg6o8XYFJT2XLHHXD4MNdFfca+fbB5s79DKaVUYKroIbKpQGcAEWkCfATEYx06e6Z6owWmg+edB3XrkrphOqDdlZVS6mQqWmDaAqtcjwcDy40xlwE3Y926+KxngoNh6FBqf/MxzaMP6Il+pZQ6iYoWmCCg0PW4L7DA9Xgz1k2/aobhwxGHg9GNZ+sejFJKnURFC8xa4G4R6YVVYD53zW8E7K/OYAEtJQVSUrj6yHTWrYPDh/0dSCmlAk9FC8zDwB1ABjDLGPOza/6VwPfVmCvwDRtGw50rOIdfyMz0dxillAo8FSowxphvsa6BSTDG3Oa2aDJwd3UGC3hDhmCCgrhVput5GKWU8qDCw/UbY4qxRlDuICLtRSTcGLPVGLPXC/kCV2IicumlDAueQeaSYn+nUUqpgFPR62CCReQF4BCwBvgZOCQiz9eY+8K4Gz6ceo5dRGZ+RVGRv8MopVRgqegezPPATcBdQGvAjnVo7GbgueqNdga44goKIuP4S8F01qzxdxillAosFS0wQ4ARxpjpxpjNrmkacDswtNrTBbqwMByDbuQa5rEy/Yi/0yilVECpaIGJwbrmpazNQGzV45x5ov7fMCLIRz54399RlFIqoFS0wKzBuqtlWfe7ltU8557Lzui2dP5pur+TKKVUQKlogRkDDBOR9SIy3TWtxzovM7r6450BRNjWexjnFi5l9+JN/k6jlFIBozLXwbQG/gtEuaYPgIvxvGdTI9S+52aKsXHw3+/4O4pSSgWMCt0PBsAYswv4P/d5IpICDKquUGeac/o1YlFQPzp/+Q44nwRbhS8vUkqps+CWSSEAACAASURBVI5+E1aD4GBY3mYYdbK3wTff+DuOUkoFBC0w1aR44NUcpTaOt/Rkv1JKgRaYanNeWi3e53pk7n8hJ8ffcZRSyu/KdQ5GROafpkl0NWQ5o51/PlzJMG7PnwJz58Itt/g7klJK+VV592AOnGbaAtToLlSxsXCofU92R7SA6XqYTCmlyrUHY4y51dtBzgY9LhDe3jyMRxc9iWzbBk2b+juSUkr5jZ6DqUYXXABv5N+CGAPvvuvvOEop5Vc+LTAiEi8i80QkV0S2iciQk7QTERknIgdc0zgREdey1iLykYjsE5GDIvKFiLTx5fs4mR49YBvNWJvQm7w33gFj/B1JKaX8xtd7MBOBQiARa/Tl10WkvYd2I4GrgRQgGRgI3OlaFgvMB9q41vM98JF3Y5fPXtct1ybsH0bE9o38/KbeS1kpVXP5rMCISCTW1f6PGWNyjDFLsArFzR6aDwMmGGN2GGN2AhOA4QDGmO+NMVOMMQeNMQ7gJaCNiNTxyRs5hePXWP6XweRSiyK9JkYpVYOJ8dFhHBHpDCw1xtRymzca6G2MGVim7RFggDFmuet5N2CRMaa2h/VeDbxujGlwku2OxNojIjExsevs2bOr9D5ycnKIioryuGzdumgeeCCFwkIb7zCM6yPmsXzef3GGhVVpm5XJ4g+BlEezeBZIWSCw8mgWz/r06fODMaZbpV5sjPHJBPQC9pSZdweQ4aFtMdDW7bkdMLgKotv8xsBO4MbyZOjataupqkWLFp1y+bJlxvTpY8xFfGUMGDN7dpW3WdksvhZIeTSLZ4GUxZjAyqNZPANWmkp+7/vyHEwOf74gMxrILkfbaCDH9WYBEJG6wELgNWPMrGrOWmmpqfDxx7CxUR/2hDTBvD3N35GUUsovfFlgNgDBImJ3m5cCrPPQdp1rmcd2IhKHVVzmG2P+6YWsVRIZCeNesDHFcTNm4ULYtcvfkZRSyud8VmCMMbnAXGCsiESKyAXAVYCnC0beAR4QkUYi0hB4EJgGICLRwBdY53Me8Un4SrjhBljb+RZsxkn+lJn+jqOUUj7n627K9wARwF5gFnC3MWadiPQSEfcRIicDHwM/A2uBT13zAK4BzgVuFZEctynJZ++iHERg9JttyOR8Dr8yXa+JUUrVOD4tMMbqWny1MSbSGJNkjHnPNX+xMSbKrZ0xxowxxsS7pjHHz78YY6YbY8S1jii36Q9fvpfy6NoVNvYYTv396/jjf6v8HUcppXxKh4rxsoun/oV8wvjpgWn+jqKUUj6lBcbLEtvEsjX5Ks7fOouFnxT6O45SSvmMFhgfaDl2GAkc4OO7PsXh8HcapZTyDS0wPhBy+QDy4+rTd+d0XnvN32mUUso3tMD4QnAwYbcO5Qr5lFce28e+ff4OpJRS3qcFxkdk+DCCTREDc2bx+OP+TqOUUt6nBcZXOnaELl14MGEab7wBa9b4O5BSSnmXFhhfGjaMJvt+pEftnxk1Sq+9VEqd3bTA+NKNN0JwMK92m05GBsyd6+9ASinlPVpgfKluXbj8clJWTeWVxH8y895M8vL8HUoppbxDC4yvnX8+cugQ9+59nBl7+jJnlN5WWSl1dtIC42tFRQCIcRJGIVvezmDnTj9nUkopL9AC42t9+4LrFso2cfKzac8jAXvTAaWUqjwtML6WmgqLFsFddyHh4UyKeZiFM7LI1CNlSqmzjBYYf0hNhddfh4ULqZv3B98E9+WJe/bhdPo7mFJKVR8tMP7UsyfyySe0ks08v7o/c14/6O9ESilVbbTA+FufPtjmf0Q7+ZV2owaQvf2wvxMppVS10AITAGyXDGDL+LmcU/QTh86/FLKz/R1JKaWqTAtMgGjzwOW8nvY+DXatJO+iyyA319+RlFKqSrTABJDrZl7NbWHvEbpyGQwcCMeO+TuSUkpVmhaYANKwIbR74jpu4R1MRgZccw3k5/s7llJKVYoWmADzt79BZvOhPFxnCixcyKG+g6Gw0N+xlFKqwrTABJjwcBg5El7Yfyt3Mom4ZZ9ysP9fwOHwdzSllKoQLTAByOkEEXiDO7mPV4j/9n9w000l45gppdSZINjfAdSf9elj7ckUFsJE532EmkImvD+aIlsowTOmQVCQvyMqpdRp6R5MAEpNhfR0ePpp+PprkAcf5O88S/DsGey75g50TBml1JlA92ACVGqqNQGkpcGiyx/lpasL+NvHT7Gyeyidlr3u13xKKXU6WmDOEH36QOetTzC/VyFXrnyO9PoHqNM3wRr6/3glUkqpAKKHyM4gsXHClT//k209bqDvwf+S/MFkinr3xSzTsf6VUoFHC8yZRoSml3fEINgwBDnyWHHNP9n3u45fppQKLFpgzkR9+iAR4RgRjNg4b++nhLZqwqZrx8COHf5Op5RSgBaYM5Orm9mWESOwLV3C77OW813MJTSb9yLFSc1x/GUorFrl75RKqRpOC8yZKjWVP4YOhdRUWtxwHml7ZjPhrk28Yu6j4IOPoWtXq2fAJ59ot2allF/4tMCISLyIzBORXBHZJiJDTtJORGSciBxwTeNERNyWvyEi60XEKSLDffYGAlhYGDz8ejO6fvMiqY2385C8wMHvN8HAgeQ1bweTJ0Nenr9jKqVqEF/vwUwECoFEYCjwuoi099BuJHA1kAIkAwOBO92WrwHuAfQ4UBkXXghLfo5h3SWjSTz2OzfyHr9sj4K77oKkJHj8ccjK8ndMpVQN4LMCIyKRwCDgMWNMjjFmCTAfuNlD82HABGPMDmPMTmACMPz4QmPMRGNMOqBj2XsQEwO9eoHTFsJsbqSbWUEaGSyjB+aZZzBNm8Ltt8O6dZCZCc89Z/1USqlqJMYY32xIpDOw1BhTy23eaKC3MWZgmbZHgAHGmOWu592ARcaY2mXaLQHeMsZMO8V2R2LtEZGYmNh19uzZVXofOTk5REVFVWkd1eVUWdati+bBB1NwOITgYMNFF+3lp59iidy1nQdtLzKMdwh35mHEBhicISGsefFFjrb3tENZ9Ty+plk8C6QsEFh5NItnffr0+cEY061SLzbG+GQCegF7ysy7A8jw0LYYaOv23A4YXAXRbf4SYHh5M3Tt2tVU1aJFi6q8jupyuizLlhnz7LPWT2OMcTqNWb7cmFGjjGmfuM98QT/jBGPAOMEUJzU15sknjVmyxJjCwmrP40uaxbNAymJMYOXRLJ4BK00lv/d9eQ4mB4guMy8a8HSFYNm20UCO682qckpNhUcfPTGSjAicdx689BKs2ZlAnVfH4giKoAgbRQSzcUctnE8+BT17YurUgSuvhFdfhd9+s8qQUkpVgC/HItsABIuI3Riz0TUvBVjnoe0617LvT9NOVVJQEHS9NxW6plOUnsHysDQmrkpl8UcH6Z63iKudX3Lp4q9I+PhjAI7GNCb/gn7UG9IP+vWDxEQ/vwOlVKDzWYExxuSKyFxgrIjcDnQCrgJ6eGj+DvCAiCzAOjT2IPDq8YUiEorVQUGAEBEJBwqNMXrBR0WlphKcmkpPoCeQmxvP/PmDmDVrECM+h0ZsoT9f0e/Il/RdMB8WTLNe17Ej9O9vFZsLL4SffiJp5kwdfFMpVcLX3ZTvASKAvcAs4G5jzDoR6SUiOW7tJgMfAz8Da4FPXfOOWwjkYRWnN1yPL/R+/LNfZCTceCPMnw979kDna5rzJnfwF96nLvvoykqej/8X6/bVo+iViXDZZZjYWOjZk+ZvvWXdW+C99/Tum0op3xYYY8xBY8zVxphIY0ySMeY91/zFxpgot3bGGDPGGBPvmsa4n38xxqQZY6TMlOHL91ITxMfDQw9BRIR1SC00zEbq/+tK5oUP06f4K2oXHaI/C1lmemCcTgSs23AOHQq1a8O551rdoV99FRYvhiNH/P2WlFI+pPeDUad0/O6aGRnWzsnxo1/GwMaNESxd2p9v5kbR5dO+BJtCigjhBRlDy9hczt27hqYffETYlCkl68tv0Izw7p0gJcWaOnWCZs2sHgiZmX/ekFLqjKUFRp2W+901jxOB1q2tiVtTITOdX16bxsHk4Rw7kMqkpTBiBRQUGBqwm86sJpk1dN6zhn7LVhP30UfI8Z3S6GiryPzyCxQXQ2gofPEF9O7t67eqlKpGWmBU9UhNZW9BAWlpVocBgIICWLVKePrphiz4rCELuMzqsrEXapFLz5i1DKi/hvNC19BpxydEFRVZh9kKCuCii6yOBJ07n5g6dbIOvSmlzghaYJTXHO9Q9thj1pGvwkIICYHnnwenM5Kff+7O+z935/G1kHzsJtLpSyiFOAliSYO/0Dh7L0nzFhA+bVrJOvOa2NkS25nICzrT9GpX4alXz2/vUSl1clpglNed7DzOcU4nbNmSyooP0ilOzyDdmcbcXals3AjFxdYhtu7Bq+gd/SNNt/9Ip+3f0/Tn92GSawWNGpXe0zGGpM8+0y7TSvmZFhjlE57O4xxns0HLltDykVR4JJWLgH8C+fnw22/Czz835OefG/La/65g40HrNbEcohOr6Wb7kV7Zq+j0zY80/nQBNtelUM0BpkyxzuN07w52O7RqZU0NGlgbVUp5lRYYFbDCw63TLp06Wc+vuQb69rUOteUFx9FrTB+Kivow7TdrNJsdG47xr+LR3MkkgjA4jeHokrVEfbuUYKejZL0mIoK8Bi3ZHt6KqM52Gl3Y6kTxadz4RPHRXm1KVYkWGHXGON2hNoejFrvn3gy3TKPYUUiRLZQxbeczd8d5RB3ejp2NtGITbQo20fz3TbRiA3V++QxmFpSsw4SFIS1aQJ068N13Vq+2kBB44QWr40FiorVM94CUOi0tMOqMcqpDbSEhkPSXVEhK5/epU2lx2228kZrKZAP79jVj/fpm/PZbf955B5YssV4jOGnETlqxySpABZtot3kT529cRp3jvdoKC+H++09sKCgIEhPJqV2fLJNIbXt96iXXh/quKTHxxON163QIHVVjaYFRZ5/UVP4oKKCF6wtdxOpoVq+edSO2Dh1OHGoLDbUx6YMmJCQ0YevWPmzZAp9shfTVmTz7/YmLR//GixwggfrsoUXYHprkZhGxaw+J7CFsw08Uf5ZFkNPz8DjNAaZOtcZu69oVmjSx7i56/GdMjM9+NUr5khYYVeOc7FBb9+6lWkFmOs5FGRzpmMawhFS2boWtW+G3rfDa17DJbeQbcTqJ5yCJZFGfPTSy7WFYyEz6FHyODYNxOsnPWE7Yl19hcxaXylMUGc2hyCYEt0wirqOr6LgXoEaNYNUqPR+kzjhaYFSNdKpDbe6NbKmpJAAJlG6fmem+FwRTp9po0CCBPXsS2LOnPVlZ8N1PLbjgswyCnYU4CKW/YwGZzvOozx6S+IMmbLd+5m4nKfcPmuzdTosVK4kv2venKCUD8dlsmPNTsbVsAbGxpab1WbH8sCmGDj1jSb7QNT8mBoKDS0Lr4TrlS1pglKqE03U4cLWCzBPng77tnsrBg7BnTyPXlMp778GLn5+4n1vj+lA7OI+wfTuIz7WK0FBm0JevsWFwOp3sXvY7jsxdxMphop2HsbnKTxvXxNTSKZy1ojC1IrAd2E9zY6zDdZdfDu3bQ0JCqWnVHwlkrE0g9eJoUnuI5zevvetUOWmBUaqSyrsXdPx8kI0T3+UdOliLW7Y8McpBaCi8/z6kpkYAdgoL7Rw4AMfS22BuX4bTUUhxUChfDPuQNbVS2b8fDuxzkpuVw6Eth7FlHyYWD9Oxw1x47Fu6sA8bYJxOcj/NIOyTzwkxjlJxu7gmx9hgcmsnIHUTcNZJwFY3gaDEBEKK87HNmmn1rgsOhpdftk5sxcZCXJw19LZI+WqQ7lGd9bTAKOVHp9oTCg21rgnlplRoaTWypaVxW6kvYxsQTWZmNH37JpUMxzN1qjV+6OHDcOgQbFuVScorfTFFhRTbQvlnzy9YLufjOJiNHNhPyJH9hOXsJwG3Kds1/b6fBNaVzC85YFdYCHffXer9OGyhHAuJJb4glj7EcVji+KldLFGN4yA2FomPI6hOLBE5e0l47WmaOwox776LTJ1qnQQLDy+ZMn8MJ2Nx0GmLVJX2poyxhpJYvJikGTO02FUzLTBK+Vl594RO1ei0h+yGpMKgE4frnitpEO2aWrBkCQwYYNWN4zsntZrCnqOwMRuysyH656Xc/E4/bMUOiiWYaW3Hscs0IDjnMGHHDhGWd5jwvENEc5g4DhFnDhK+bjNR66znIXjoaVdQYN1DqOx7AroRTAFhHAkJp7jMFGwKqZu1DjFOjNg4kpSMiYgg2OnAZooIcjqwFTsoOubAkV9ERLCDUHGAw4EUFYHDenxcyegPdju0aGF1N69Xz/qZmMgvBxLJ/D2RlP716HZp3RPnttxVx+HDs2jPTguMUmeJ0xaqMt23y+rZszznlS6A27+GjAyC0tIY6aGReweIkBCYPt26rcPGHEP+wWMUZh2i9qoMzn/rdmxFDpxBwaT3eJyssCSKj+VjjuWzf0c+2fvzCSefMAqIteVTi3yC8/IJyc4nxJlPW36jHtaN7oxxcnDbEX4nAQchOAihiOCTPi62hUBwMCYihHOLM+lZ6DrHhfDHdhv5ew4Q7/iFuMIsQpyFALRzTbwKToT8yDrkRSdSEJOII64eQTYnDb+bhziLMEHBHLzhHmjVipDwIELDgwiJCGLrH0H8uiGINu2DaX1OkHVNlfu0fj08+ijNCwvh3Xdh0iQ47zyr2ByfwsP57scwFi0JIa2PeG/vrhpogVFKlfDu3pQAka7pJrilZcke1cVl1le2l156eulNOp3g+DYTLuuLKSyEkFB4cyaxbVLJy6Nk+mC2dV7LGOt6qP79rfUUFFhj3RUUQMH2TLp/toygYuvw4Ysd3mZ1RKrVJs8QkncUsrKIyM6iHntJJMuacrOol7uXxN1ZJLKC+mzHhrVHJEUOEma8/KffTSvXxAen/hWX3Lbi1ls9Lj/fNeUTRl5oGCYkjOLQcJzBYThDwrA5HURnbQLjBJuNA8l9KKyfBBG1oFYtJNL6uetwLXYcrMU5XWvRupM1r9T06680gvqnTntyWmCUUtWuoh0gPL3+VHtTNhuEpZ1oJGlptExNpWWZ9SQmwvz5JwrVk096ylW6t98rpRoIEENmZgx9+7YuWc/ChdYYeccL2bFjsPW7TFrd1RdxFGKCQ1ny4Dx21e9CwbFiCo4V882iYpYvKcJGMcEUk9KhmFbNiykuPDE1ObiGe377K0HGOgQ5sd5TbLM1J8hRgM1RQFBRASa/gKDigpK9u7BC15RbYD2ngPasozbOkk4dztU/4WAjtThGLY4RyTEAGgBdAeaf/M9UHxqd5i95UlpglFIBySfnptwanurw4cnWExXl1qhtKrQ50ah3mXUlZ8L/3K+desNTnu6Q2bGk2P3tNIcgQ0NhwQLo0sU6nVRYaP20Lc+EW/piHIU4g0PJeuUjDrZJLTntVFhgmDcrn08/OEa4OUZt2zFG33OM2244ZlXLY8dg1qwTu3+VpAVGKXVWK1ehqq71nKKRt4tdKUmp0Di95FxZxz81EuolRvD+xxEcKqzDgVA4ZwhWz4rj6tWD+fMxeXmVrjBaYJRSykcCpdgdX3zKQuVqkNWjx67K5tQCo5RSNVR5eh7uhD2VXb/e1EIppZRXaIFRSinlFVpglFJKeYUWGKWUUl6hBUYppZRXaIFRSinlFVpglFJKeYUWGKWUUl6hBUYppZRXaIFRSinlFVpglFJKeYVPC4yIxIvIPBHJFZFtIjLkJO1ERMaJyAHXNE5ExG15JxH5QUSOuX528t27UEopVR6+3oOZCBQCicBQ4HURae+h3UjgaiAFSAYGAncCiEgo8BEwA4gDpgMfueYrpZQKED4rMCISCQwCHjPG5BhjlmDdR+1mD82HAROMMTuMMTuBCcBw17I0rFGg/22MKTDGvIJ127mLvPwWlFJKVYAvh+tvDRQZYza4zVsD9PbQtr1rmXu79m7LfjKm1G3WfnLN/7zsikRkJNYeEUCOiKyvXPwSCcD+Kq6jugRSFgisPJrFs0DKAoGVR7N41qayL/RlgYkCjpaZdwSofZK2R8q0i3Kdhym77FTrwRjzBvBGZQJ7IiIrjTHdqmt9VRFIWSCw8mgWzwIpCwRWHs3imYisrOxrfXkOJgeILjMvGsguR9toIMe111KR9SillPITXxaYDUCwiNjd5qUA6zy0Xeda5qndOiDZvVcZVkcAT+tRSinlJz4rMMaYXGAuMFZEIkXkAuAq4F0Pzd8BHhCRRiLSEHgQmOZalgEUA38VkTARudc1/2tv5ndTbYfbqkEgZYHAyqNZPAukLBBYeTSLZ5XOIqXPlXuXiMQDU4H+wAHgEWPMeyLSC/jMGBPlaifAOOB210vfAh4+fmJfRDq75rUDfgVGGGN+9NkbUUopdVo+LTBKKaVqDh0qRimllFdogVFKKeUVWmDKwdWZYIpr/LRsEVktIpcGQC67iOSLyIwAyHKDiPzqGmdus+u8mj9yNBORBSJySET2iMh/RMQn13uJyL0islJECkRkWpllfUXkN9f4eYtEpKk/sojI+SLypYgcFJF9IvKBiDTwR5YybR4XESMi/byZ5XR5RKSWiLwmIvtF5IiIfOvHLNe7/k1li8gvInK1l7Oc8nuuMp9hLTDlEwxsxxp1IAb4B/C+iDTzYyawxnZb4ecMiEh/rE4Zt2Jd8Hoh8Luf4rwG7AUaAJ2w/mb3+Gjbu4BnsDqylBCRBKwelI8B8cBKYI4/smCN3/cG0AxoinX92Nt+ygKAiLQErgN2ezlHefK8gfU3Osf182/+yCIijbDGW3wA6zq/h4D3RKSeF7Oc9Huusp9hX17Jf8ZydbF+0m3WJyKyBegKbPVHJhG5ATgMLANa+SODm6eAscaY71zPd/oxS3PgP8aYfGCPiHzOiWGGvMoYMxdARLoBjd0WXQusM8Z84Fr+JLBfRNoaY37zZRZjzGfu7UTkP8A33shwuixuJgIPY/3nwOtOlkdE2gJXAo2NMcdHHfnBH1lcjw+7/b0+FZFcoCXWf6C8keVU33N1qMRnWPdgKkFEErHGVvPLxZ0iEg2MxfrfjV+JSBDQDagrIptEZIfrsFSEnyL9G7jBdaijEXApHsao87FSY+u5/iFvxkeF7zQuxI8XKYvIdUCBMWaBvzK4OQ/YBjzlOkT2s4gM8lOWlcCvInKliAS5Do8VYI276BNlvucq9RnWAlNBIhICzASme+t/n+XwNDDFGLPDT9t3lwiEAIOBXliHpTpj7V77w7dYH/qjwA6sf6j/81OW4yo0fp6viEgy8DjW4Rd/bL828Cxwvz+270FjoAPW36YhcC8wXUTO8XUQY0wx1gXn72EVlveAO11f7F7n4XuuUp9hLTAVICI2rJEHCrE+fP7I0AnoB7zkj+17kOf6+aoxZrcxZj/wInCZr4O4/j6fYx0rjsQakTYO6/yQPwXc+Hki0gr4DLjfGLPYTzGeBN41xmz10/bLygMcwDPGmEJjzDfAImCAr4O4Ojs8j3V7klCs8yJviQ9urniS77lKfYa1wJSTiAgwBet/7IOMMQ4/RUnDOkH7h4jsAUYDg0RklT/CGGMOYe0puF+x66+rd+OBJKxzMAXGmANYJ7B9XuzKKDW2nlj3RmqJ/w6xNgW+Ap42xngaqslX+mIN+bTH9VlugnVS+WE/5fF0+Mlfn+VOwLfGmJXGGKcxZgWwHOs/l15ziu+5Sn2GtcCU3+tYPUsGGmPyTtfYi97A+sN2ck2TgE+Bi/2Y6W3gPhGpJyJxWD1vPvF1CNfe0xbgbhEJFpFYrJvX+eS4tWub4UAQECQi4WJ1kZ4HdBCRQa7lj2Pd08hrh1hPlsV1XuprrCI8yVvbL08WrALTgROf5V1Yd66d6Kc83wJ/AI+62lwA9AG+8EOWFUCv43ssYg2P1Qvvf5ZP9j1Xuc+wMUan00xY3TkNkI+1q3h8GhoA2Z4EZvg5QwhWD6DDwB7gFSDcT1k6YQ2Iegjrhk3vA4k+/FuYMtOTrmX9gN+wDsNkAM38kQV4wvXY/XOc46/fS5l2W4F+fv47tQcygVzgF+AaP2a5F9iEdRjqd+BBL2c55fdcZT7DOhaZUkopr9BDZEoppbxCC4xSSimv0AKjlFLKK7TAKKWU8gotMEoppbxCC4xSSimv0AKj1BlCrPulDPZ3DqXKSwuMUuUgItNcX/Blp+9O/2qlaia9H4xS5fcVcHOZeYX+CKLUmUD3YJQqvwJjzJ4y00EoOXx1r4h86rql7DYRucn9xSLSUUS+EpE8sW5ZPE1EYsq0Gea6D0mBiGSJyPQyGeLFus1xroj87mEbj7u2XeAaRPIdr/wmlCoHLTBKVZ+ngPlY46G9AbzjulPh8dFnv8Aa2+k84BqgB263yhWRO4HJWIOHJmONAr22zDYeBz7CGtl2DjBVRJJcrx+ENbr2PYAduAL43gvvU6ly0bHIlCoHEZkG3IQ1EKC7icaYh0XEAG8ZY+5we81X8P/bu58QG6M4jOPfX5Gys5FiKxFLERtFWdgNsrFgYbbCgtnIRlEmdvZiRRaKsrCxMEnWd+HWLJj8udI0apoRHovfmWl6o3nveI+xeD5169zz3k7nLt5+vec9nYcPkk5GxBngBhnH+7VcP0DmjWyV1I+Id+TBpZf+MAcB1ySNle9ryGC1UUl3I+I8eRrxTq1enITZIr+DMWvvOTDa6Jte0p5oXJsAjpT2dvJ486UBTS+An8COiJgBNgPPlpnD4nHtkr5HxADYWLruk+mQkxHxlAxfeyRpfpkxzarwEplZe7OS+o3P5w7GHWYZoflkIsp9LOktsI18ipkBxoHXZXnO7J9zgTHrzt7ffO+Vdg/YVXLoF+wj78GepE/AFBnCtWKS5iQ9lnQO2E3mm+z/mzHNVspLZGbtrYuITY2+H5IGpT0SEa/IMKZjZLHYU67dIzcB3ImIy8AG8oX+Q0n98purwM2Ih5zuTwAAAK9JREFU+EimlK4HDkoabzO5iDhF3tMvyc0EJ8gnnjdD/k+zTrjAmLV3CHjf6JsCtpT2FeAomeg5AE4rs9SRNBsRh4Fb5M6uOXI32NmFgSTdjohvwAXgOvAFeDLE/KaBi+RmgrVkIuOIpMkhxjDrjHeRmXWg7PA6LunBas/F7H/hdzBmZlaFC4yZmVXhJTIzM6vCTzBmZlaFC4yZmVXhAmNmZlW4wJiZWRUuMGZmVsUv8IlMDT1z2zEAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "def plot_learning_curves(loss, val_loss):\n", " plt.plot(np.arange(len(loss)) + 0.5, loss, \"b.-\", label=\"Training loss\")\n", " plt.plot(np.arange(len(val_loss)) + 1, val_loss, \"r.-\", label=\"Validation loss\")\n", " plt.gca().xaxis.set_major_locator(mpl.ticker.MaxNLocator(integer=True))\n", " plt.axis([1, 20, 0, 0.05])\n", " plt.legend(fontsize=14)\n", " plt.xlabel(\"Epochs\")\n", " plt.ylabel(\"Loss\")\n", " plt.grid(True)\n", "\n", "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "HhHs0TEs5jk7", "outputId": "8ec7db41-b8ce-4486-8629-5cf8ff537d14", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVyVVf7A8c/3sgqoIAIuKKgI7hsuqJloaqvltFhp2zRl0zLNTNM201ROU1NTTb9mKVumpqmsxkpbJ7eEXHIDFRXBDUFFFlkE2QTuPb8/7oW5ICDLXeG8X6/npfc85zn33Id77/c+55znHFFKoWmapmn2ZnB2BTRN07SuQQccTdM0zSF0wNE0TdMcQgccTdM0zSF0wNE0TdMcQgccTdM0zSF0wNE0TdMcwiUDjog8ICJJInJORN67QN5fi0iuiJSKyLsi4mO1L1JEEkSkQkTSRWSO3SuvaZqmNcklAw5wCngWeLelTCJyKfA4cAkQAQwG/mCV5WNgNxAMPAF8JiIh9qiwpmma1jJx5ZkGRORZIFwpdUcz+z8CMpVSv7M8vgRYrpTqIyLRwD6gt1LqrGX/Jsv+NxzyAjRN07R6ns6uQAeNBL60epwChIlIsGVfRl2wsdo/sqmCRGQJsATA19c3duDAgfapsRsxmUwYDK56Eew4+jzoc1BHn4eWz8GhQ4cKlFLNtiK5e8AJAEqsHtf9v3sT++r292+qIKXUW8BbADExMergwYO2rakbSkxMJD4+3tnVcDp9HvQ5qKPPQ8vnQESyWjrW3UN1GdDD6nHd/882sa9u/1k0TdM0h3P3gJMKjLV6PBbIU0oVWvYNFpHujfanOrB+mqZpmoVLBhwR8RQRX8AD8BARXxFpqvnvfeBnIjJCRAKB3wPvASilDgF7gKctx/8EGAN87pAXoWmapjXgkgEHc+CoxDzk+RbL/38vIgNFpExEBgIopVYDLwIJwHEgC3jaqpybgIlAMfACcL1S6rTDXoWmaZpWzyUHDSillgJLm9kd0CjvK8ArzZSTCcTbrmaapmlae7lkwHF1paWl5OfnU1NT4+yq2FXPnj1JS0tzdjUcxt/fn/Dw8C4/7FXT7EUHnDYqLS0lLy+P/v37061bN0TE2VWym7Nnz9K9e/cLZ+wETCYT2dnZFBQUEBoa6uzqaFqnpH/KtVF+fj79+/fHz8+vUwebrsZgMBAWFkZJSeNbtzRNsxUdcNqopqaGbt26Obsamh14eXlRW1vr7GpoWqelA0476Cubzkn/XTXNvnTA0TRN0xxCBxxN0zTNIXTA0TRN0xxCB5wuQERa3O644w6n1S0yMpKXX37Zac+vaZrj6IDjJMuXQ2QkGAzmf5cvt99z5eTk1G9vv/32eWl//etf21RedXW1PaqpaVonpwOOEyxfDkuWQFYWKGX+d8kS+wWdPn361G+BgYEN0srLy7ntttvo06cP/v7+TJgwgW+++abB8ZGRkSxdupQ777yTwMBAFi9eDMC7777LwIED8fPzY/78+bz++uvnjfT6+uuviY2NxdfXl0GDBvHEE0/UB6z4+HiysrJ45JFH6q+2NE3rvHTAcYInnoCKioZpFRXmdEcrKyvj8ssvZ926daSkpHDddddx7bXXkp6e3iDfK6+8wrBhw0hKSuJPf/oTW7du5a677uL+++9nz549XH311Tz99NMNjlmzZg2LFy/mgQceIDU1lXfffZfPPvuM3/3udwCsXLmS8PBwnnrqqfqrLU3TOjGllN4abdHR0ao5Bw4caHZfa4koZb62abiJdLjoC/r000+V+c/evClTpqg//vGPqrS0VCmlVEREhLrqqqsa5LnpppvUpZde2iDt7rvvblD2jBkz1DPPPNMgz6pVq5S/v78ymUz1Zb/00kvtfj221tTfNyEhwfEVcTH6HJjp89DyOQCSVAvfrfoKxwkGDmxbuj2Vl5fz6KOPMmLECIKCgggICCApKYnjx483yDdx4sQGj9PT05k8eXKDtClTpjR4nJyczHPPPUdAQED9tmjRIsrLy8nNzbXPC9I0zWXpyTud4LnnzH021s1qfn7mdEd7+OGHWb16NS+//DJDhw7Fz8+P22677byBAf7+/m0u22Qy8fTTT3PDDTecty8kJKTdddY0zT3pgOMElj53nngCjh83X9k899z/0h1p8+bN3HbbbVx33XUAVFVVcfToUaKjo1s8btiwYezcubNB2o4dOxo8njBhAunp6URFRTVbjre3N0ajsZ211zTNneiA4ySLFzsnwDQWHR3NqlWruOaaa/Dy8uIPf/gDVVVVFzzuwQcf5KKLLuKll15iwYIFbNy4kVWrVjXI89RTT3HVVVcRERHBwoUL8fT0ZP/+/ezYsYMXX3wRMI+A27RpE7fccgs+Pj707t3bLq9T0zTnc9k+HBHpJSKrRKRcRLJEZFEz+b6zLDtdt1WLyD6r/ZkiUmm1f63jXoXre+WVVwgNDWXGjBlcfvnlxMXFMWPGjAseN3XqVN5++23+9re/MWbMGL744gsee+wxfH196/NceumlfPvttyQkJDB58mQmT57MCy+8wECrzqpnnnmGEydOMGTIEN3MpmmdnCtf4bwGVANhwDjgWxFJUUqlWmdSSl1u/VhEEoENjcqar5Rab8e6uo3rr78e82ASs4iICNavb3hqHn74YcC8ABtAZmZmk2Xdeeed3HnnnfWPf/3rX5/XfDZv3jzmzZvXbH3i4uJISUlp02vQNM09uWTAERF/4DpglFKqDNgsIl8BtwKPt3BcJDADuMP+tdReeukl5s6dS0BAAOvXr+eNN97gT3/6k7OrpWmai3LJgANEA7VKqUNWaSnAzAscdxuwSSmV2Sh9uYgYgN3AI0op/ZPaBpKSknj55ZcpKSlh0KBBPP/88/zyl790drU0TXNRYt284ipEZAbwqVKqj1Xa3cBipVR8C8cdAZ5VSr1nlTYd2AUI8EvLNkwpdabRsUuAJQAhISGxK1asaPI5evbs2eKoq87EaDTi4eHh7Go41JEjR85bZrqsrIyAgIAGaYeLa0krNDEi2IOooM5/jpo6B12RPg8tn4NZs2YlK6UmNrkT173CKQN6NErrAZxt7gARuQjoA3xmna6U2mL18HkRuR1zs9vXjfK9BbwFEBMTo+Lj45t8nrS0NLp3796qF+Huzp4922Veax1fX1/Gjx/fIC0xMRHr90NyVjHPr/kRk4L/ZhlZflccsRFBDq6pYzU+B12VPg8dOweuOkrtEOApIkOt0sYCqc3kB7gdWGnp82mJwny1o2ntsmrXSUyWhoFzNSa2ZRQ6t0Ka5iZcMuAopcqBlcAzIuJvaRa7Bvigqfwi0g1YCLzXKH2giEwXEW8R8RWRR4DewJYmitG0Vkk5+b/WWAUM79u1rgI1rb1cMuBY3Ad0A/KBj4F7lVKpIjJDRBpfxSwAzgAJjdK7A8uAYiAbuAy4XCmlf5Jq7bLlSAH7skv56fRI7rxoEB4Ca1PznF0tTXMLrtqHg1KqCHMgaZy+CQholPYx5qDUOG8qMMZeddS6FqUUL605SN+evjx22TB8vTwwAO9sOcYtcRGM6t/T2VXUNJfmylc4muZS1qfls+fEGX55yVB8vcwj0x6cM5Reft4s/SoVVxzxqWmuRAccR3rxRUho3OrXSEKCOZ+b+uyzzxqs3Pnee+91eBhpYmIiIkJBQUFHq9duJpPi5TUHGdTbn+tiw+vTe/h68cilMSRlFfNVyimn1U/T3IEOOI40aRIsXNh80ElIMO+fNMnmT33HHXfUL+Ps5eXF4MGDefjhhykvL7f5c1m78cYbycjIaHX+yMhIXn755QZp06ZNIycnh+DgYFtXr9W+3nuKg3ln+fXcaLw8Gn5sbpg4gFH9e/DCd+lUVNc6qYaa5vp0wHGkWbNgxYqmg05dsFmxwpzPDubMmUNOTg4ZGRk8++yzvP766/Xzplmrra21WfNQt27dCA0N7VAZ3t7e9OnTp8GVkyPVmhSvrDvEsD7duWp03/P2exiEpfNHklNSxRuJR51QQ01zDzrgOFpTQccBwQbAx8eHPn36MGDAABYtWsTixYv54osvWLp0KaNGjeK9995jyJAh+Pj4UF5eTklJCUuWLCE0NJTu3bszc+ZMkpKSGpT5/vvvExERgZ+fH1dddRV5eQ1HbDXVpPbf//6XKVOm0K1bN4KDg5k/fz5VVVXEx8eTlZXFI488Un81Bk03qa1cuZLRo0fj4+PDgAEDeO655xoEycjISJ599lnuueceevToQXh4OC+99FK7ztvm7FqyCit45NIYDIamg97EyF5cPbYfb27M4ERRRZN5NK2r0wHHGayDzlNPOSTYNKVbt27U1NQAcOzYMT766CM+/fRTUlJS8PHx4YYbbiA7O5tvvvmG3bt3c/HFFzN79mxycnIA2L59O3fccQdLlixhz549zJ8/n6eeeqrF51y9ejVXX301c+fOJTk5mYSEBGbOnInJZGLlypWEh4fz1FNPkZOTU/88jSUnJ3PDDTdw7bXXsm/fPl544QWef/55/vGPfzTI93//93+MHj2aXbt28dhjj/Hoo4+ydevWNp2jqhojXx6pYcLAQGYPa/lK7bdXDMMgwqOfpfBawhGSs4rb9Fya1ukppfTWaIuOjlbNOXDgQLP72uzJJ5UC8792dvvtt6srr7yy/vH27dtVcHCwWrhwoXr66aeVp6enys3Nrd///fffK39/f1VRUdGgnLFjx6o///nPSimlbr75ZjVnzpwG+3/2s58p89vK7F//+pfy9/evfzxt2jR14403NlvPiIgI9dJLLzVIS0hIUIA6ffq0UkqpRYsWqVmzZjXI8/TTT6v+/fs3KOemm25qkCcqKkr98Y9/bPa5lTr/7/v0l/tUxGPfqPe2HGvxuDqPf5aiIh77RkU+9o2K+f1/VVJmUauOc3UJCQnOroJL0Oeh5XMAJKkWvlv1FY6zJCTAsmXw5JPmfy80es0GVq9eTUBAAL6+vkydOpWLL76Yv//97wCEh4cTFhZWnzc5OZmKigpCQkIICAio3/bv38/Ro+Z+irS0NKZOndrgORo/bmz37t1ccsklHXodaWlpTJ8+vUHaRRddRHZ2NqWlpfVpY8Y0vAWrX79+5Ofnt/p5Nh8p4L0fswB4/ru0Vl2x9OlpXoBOUTftjfNG1mmaq3HZGz87tcZ9NrNmOaRZ7eKLL+att97Cy8uLfv364eXlVb/P39+/QV6TyURoaCibN28+r5wePRrPq+o6rAcWWL++un0mk6nVZX28/Xj9/2tqzXOmXWiSzouGhrAs8ShVtSYU5ok+q2tNeHvq33aapj8FjtbUAIGWRq/ZkJ+fH1FRUURERJz3ZdzYhAkTyM/Px2AwEBUV1WCrG3U2fPhwtm3b1uC4xo8bGz9+PN9//32z+729vTEajS2WMXz4cLZsaTgd3ubNmwkPD7fp7NYlleb+LQPg5WkgbvCFh2XHRgSx/O44Hrk0mhsmhrMh/TS3vbudMxXVNquXptlCcmYRf99w2KF9jfoKx5FaGo1mHXScMICgsTlz5hAXF8c111zDiy++yLBhw8jNzWX16tXMmTOHGTNm8OCDDzJt2jSef/55rr/+ehITE1m1alWL5T7xxBPMnz+fqKgoFi1ahFKKtWvXcs899+Dn50dkZCSbNm3illtuwcfHh969e59Xxm9+8xsmTZrE0qVLWbRoETt37uQvf/mLTVcbrTWa2JddQnx0CL3VGW6eM6nVSxDERgTV550+pDePfraXn7z+I+/cPpHBIV17LRXN8ZKzitmWUUjc4GD6Bfqy5UghX6dk88Mhc3Pvq3KY/7tpLFeP7W/3uugrHEfaubPlYFIXdHbudGy9miAifPbZZ8yePZu7776bmJgYFi5cyMGDB+nXrx8AcXFxvPPOOyxbtowxY8awcuVKli5d2mK5V1xxBatWreK7775j/PjxzJw5k4SEBAwG81vxmWee4cSJEwwZMoSQkJAmy5gwYQKffvopn3/+OaNGjeLxxx/n8ccf54EHHrDZ69994gwllTXcMHEAVw3xbvd6NwvG9+eju6dQUlnD/L9v5rHP9+rRa5rDJGcVc/Nb23hpzUGuX/YjU5/fwMOfprDj2P/eg0alePDjPdzzQRK7j9v3vemSK346W0xMjDp48GCT+9LS0hg+fLiDa+QcXXEBtrq/759Xp/P2xgx2PTWXXdu2dHjRrf/uzeH+j3ahAF8vg9st2qYXHjNzt/Pw8pqD/CPhSP3jWTEhPHrZMMrP1XLLO9upqTXh6WFg/ti+rDuQT0llDVMG9WLOiDCqa43EDe593vu0pXMgIm654qemOVVCej4TI4Po4dtyX1drHSv83xRC1a0cgKBpHVU3hsYg4O1p4IHZQxne1zzoZ/ldcfVNbbERQZSdq+WTHcd5PeEo248VAXD/zucJuGsBMTdf3fyTJCSYW2UeffSC9dFNaprWSPaZStJzz17wRs+2iBscXD9SzSDSqgEImtZRR/LLCPLz4qG50eddVcdGBHH/rKj6tAAfT+6aMZg7pkfWL4m8J2woA++9w2bzP+qAo2mNJKSb79WxZcCJjQjio7um4OftwfSoYH11o9ldVY2RHw6d5vLRfXlg9tBWv+emR/XGx8uAh0DykHEcX/aezeZ/1E1qmtZIQno+A3p1Y4iNR5TFRvZi2pBgMgrsO0O3pgFszSikotrI3BFhF85sJTYiqEFzW0xEEPSxGkEr0u75H3XAaQellNNmLtbsxzz9Bmw5WsCNEwfY5W88fmAQ69PyKamooaefbfqHNK0p6w/k4eftwdR2NN9aD+0HGty2EXn55fDdd+26fcNlm9REpJeIrBKRchHJEpFFzeRbKiI1IlJmtQ222j9ORJJFpMLy77iO1MvLy4vKysqOFKG5qJqaGqqMiqoaE/E2bE6zNn5AIAB7Tp6xS/maBuYfT+vT8rh4aEj96rQdNmsW3HsvkR98APfe2657BV024ACvAdVAGLAYWCYiI5vJ+x+lVIDVlgEgIt7Al8CHQBDwb+BLS3q7hIaGkp2dTUVFhV5SuBMxmUzk5eWxL78GXy9Du34Vtsbo8J6IYPf7HbSubV92CXml59rcnNYiy/yPmbfe2u75H12ySU1E/IHrgFFKqTJgs4h8BdwKPN6GouIxv8ZXLTOZ/k1EHgZmA6vbU7e6ecROnTpVP7V/Z1VVVYWvr6+zq+Ew/v7+/GtXIdOH9Lbdr8JGuvt6ER3and3H9RWOZj/rD+RhEJhlqyt1qz6bTBEif/rTTtWHEw3UKqUOWaWlADObyT9fRIqAHOAfSqlllvSRwF7V8FJkryW9QcARkSXAEoCQkBASExM7/CLcXVlZ2XmLp3Vmp8pMZBZWcnGYscHfv6yszKbvhzCvcyQdO0tCQoLb9AXa+hy4K3c5D6t2VhIVaGDvzh87XFbg7t2M+MMfOPD005wRMZ+DgAACf/c7RvzkJ+b08eNbV1hLaxc4awNmALmN0u4GEpvIOwLoB3gA0zAHnZst+54EPmmUfzmwtKXnb2k9nK6kq6398eYPR1TEY9+ok8UN1wCy9Xn4eHuWinjsG3U0/6xNy7WnrvZeaI47nIcTReUq4rFv1Js/HOl4YRs2KNW7t/lfiwbnoNF+3HQ9nDKg8Rz4PYCzjTMqpQ4opU4ppYxKqR+BvwLXt7UcTduQns+wPt3pH9jNrs8zbqBl4MAJ3aym2d76A+Zl3ueO6NPxwmw8/6OrBpxDgKeIDLVKGwuktuJYBfU3yqYCY6Rhu8WYVpajdSGlVTUkZRbbrs27BUNDu+Pv7aH7cTS7WJ+Wz5AQfwb19r9w5gt59NEL99HMmtWqaW3ARQOOUqocWAk8IyL+IjIduAb4oHFeEblGRILEbDLwIOaRaQCJgBF4UER8RKRuOuENdn8RmlvZdKiAWpOy6ewCzfEwCGMHBLL7hB6pptlWaVUN2zIKmWPL0Wk25JIBx+I+oBuQD3wM3KuUShWRGSJSZpXvJuAI5may94E/K6X+DaCUqgYWALcBZ4A7gQWWdE2rtyE9n57dvOrvk7G3cQMCSc85S2V1y4vNaVpb/HDwNLUmxdzhrhlwXHWUGkqpIszBonH6JiDA6vHNFyhnNxBr8wpqnYbJpPjhUD4zo0Pw9HDMb7DxA4OoNSn2nyphUmQvhzyn1vmtO5BHsL834we65lx9rnyFo2kOsS+7hIKyamYNa3rBN3sYVzfjgO7H0Wykxmgi4WA+s4eF4mFwzeH2OuBoXd7y7VkABPq1ewKKNgvp7kN4UDfdj6PZzM5jRZytqrVb/83y5RAZCbNnzyQy0vy4rXTA0bq05KxiPk06CcC9HyY7dPnn8QOD9BWOZjPr0vLw8TQwY2hvm5e9fDksWQJZWaCUkJVlftzWoKMDjtalfbUnm7ppKGosK3E6yrgBgZwqqSK3pMphz6l1Tkop1h3I46Ko3vh5275r/oknoKKiYVpFhTm9LXTA0bq002XnAPAQ8PI0OHQlzvH1N4DqZjWtY1btzuZkcSVDw+wzFdXx421Lb44OOFqXVVVjZMuRQqZH9eaheTHnLcFrbyP69sDLQ9itZxzQOiA5q5hHP9sLwL+2ZNqlWXjgwLalN0cHHK3LWncgj5LKGn4+c3CDtd0dxdfLgxH9euoZB7QO2ZZRSK3J3DBca7RPs/Bzz4GfX8M0Pz9zelvogKN1WSuSTtA/sBvThti+k7W1xg8IZN/JEmqNJqfVQXNvkweZ7+MS7NcsvHgxvPUWRESAiCIiwvx48eK2laMDjtYlZZ+pZPORAq6LDXfqPQvjBwZSWWPkYJ6eT1ZrHz9v89pNV4zua9dm4cWLITMTNmz4gczMtgcb0AFH66I+Tz6JUnBDbLhT6zF+gPnLQTerae1V12fz+OXDHN4s3FY64GhdjsmkWJF0gulRwQzo5XfhA+xoQK9u9PL31ksVaO2WlFlMWA/zjcSuTgccrcvZllHIyeJKFk4c4OyqICKMHxDI7uN6aLTWPslZxUyM6OUWq8fqgKN1OSuSTtDd15NLR9pggSobGDcgkKOnyymprHF2VTQ3k1NSSfaZSpdvSqujA47WpZRU1vDd/lyuGdcPXy8PZ1cHoH5m32e/OeDQqXU095eUaX6/TIzUAUfTXM5XKac4V2vixoltvGPNjpRlcp3Pkk+y+J/bdNDRWi05q5huXh4M79vD2VVpFR1wtC7l06QTDOvTnVH9XecDuvdkCWBeG93R87lp7i0pq4hxAwLxctA6Th3lHrXUNBtIyyll78kSFk4c4FIdrHGDg/G03Avk6eHY+dw091V2rpYDp0qZ5CbNaeDCAUdEeonIKhEpF5EsEVnUTL5HRGS/iJwVkWMi8kij/ZkiUikiZZZtrWNegeZqViSdwMtDWDC+v7Or0kBsRBCvLBwLwK1TI9ymA1hzrj3Hz2BSEOtGK8a6bMABXgOqgTBgMbBMREY2kU+A24Ag4DLgARG5qVGe+UqpAMs2z56V1mwnOauY1xKO2KRPY1tGAR/vOM6kiF708nfcQmutdfW4/gwO8Sc9R884oLVOUlYRIv+bddwduGTAERF/4DrgSaVUmVJqM/AVcGvjvEqpF5VSu5RStUqpg8CXwHTH1lizte0Zhdz45lb+svZghzvSv9uXw6K3t1NVYyIpq8hlO+XnjejDtoxCSir08GjtwpKziokJ604PXy9nV6XVRCl14VwOJiLjgS1KKT+rtIeBmUqp+S0cJ8Au4E2l1BuWtEygG+bguht4RCmV0sSxS4AlACEhIbErVqyw3QtyU2VlZQQE2GZ9jSPFRtKLjAzr5UFUUMvDkWtNiie3VJJTbn5vGoBrh3px1ZC2XZmUVSu+PlrNuqxa6qbGbE9ZtjwPLTlyxsiz26pYMsaHaf1sv4hWRzjqHLg6VzkPJqW4b30F0/p5cttIH4c+d0vnYNasWclKqYnNHeta7+r/CQBKG6WVAN0vcNxSzN8p/7JKW4w5CAnwS2CNiAxTSjWYS0Qp9RbwFkBMTIyKj49vb907jcTERGxxHnYcK+SFtdsxKYW3p7HFCQZrjCbuX76LnPIKPAyC0aQQEW6eM6lVfRvJWcVsPnya/LPn+CrlFOXnaokfFsKWI4XUGk14eRpaXVYdW52HC7nYpHgz9XtOqiDi42Pt/nxt4ahz4Opc5Tykniqhas1mrp42ingH90l25By4asApAxqPW+0BNNvALSIPYO7LmaGUOleXrpTaYpXteRG5HZgBfG276moteWfzsfr1OqprTWzLKGjyC7/GaOIXH+1m7YE8ls4fwejwQF5ak862jCJOteJu6uSsYm56ays1RvNzTYwI4k/XjiY6rDvJWcVsyygkbnCwy3bKGwzC3BFhfLE7m6oao8vcmKq5nrpmYVd9LzfHJftwgEOAp4gMtUobC6Q2lVlE7gQeBy5RSp28QNkK89WO5gBGk2LvyZL6E25SsOVIIWcqqhvkqzGaePDj3axOzeWpq0Zwx/RBxEYE8cHPpjBhYCC/W7mP44UV5z9BXbkmxZ9Xp9UHG4PArGGhRIeZL4pjI4KcsshaW80bEUZFtZEfjxY4uyqaC3OnCTutuWTAUUqVAyuBZ0TEX0SmA9cAHzTOKyKLgT8Bc5VSGY32DRSR6SLiLSK+liHTvYEtjcvR7GPdgVxySqr49dxoHp4XzaIpA9lxrIhLX93ID4dOA+ZVCn/1yR6+25/L768czp0XDao/3svDwF9vGg8Cv/hkNzVNLFRWXWvioRV72HGsGA+D4CHgbaeFqOxt6pBgAnw8WZua5+yqaC7MnSbstOaqTWoA9wHvAvlAIXCvUipVRGYA3yml6nqtngWCgZ1WJ/9DpdTPMff5LAOGAFXAHuBypZS+ldsBlFIs+yGDgb38uH9WVP1CZ4smD+TX/9nD7e/u4PJRfThWUE567ll+f+Vw7pox+LxyBvTy48/XjeG+5bt4ee1Bfnv58Pp9Z6tq+PmHyWw5Usgjl8YQN6gX244VuXTTWUt8PD2IjwlhfVoeRpNy6uJwmms6dcY8YeddMwZdOLOLcdmAo5QqAhY0kb4J86CCusfNnnWlVCowxi4V1C5o+7EiUk6c4Y8LRjX44hzVvydf/+IiHv40hW/25gDgaZD6SSybcgI6FtsAACAASURBVMXovtw8eSBv/pDB9CG9uTg6hLzSKm5/dwdH8sv4yw1juc6ymJo73QjXlLkjwvhmbw57ThQTG+Her0WzvSRL/81EN3xvuGSTmtY5vPHDUXoHeDe5qqavZcLBujiklLrgHGJPXTWC6LAAfvHxLh77PIUr/7aJE0UVvHvHpPpg0xnMGhaKl4foZjWtScmZRfh5ezC874UG7boeHXA0u0jLKSXx4GnumBbZ7GiruMHBeHsa8BDwakWfSzdvD+6Lj6Kkspb/7DxJQVk1S68eycXRIfZ4CU7Tw9eLuMHBrEnNxRXvk9OcKymrmHEDAvF0kwk7rblfjTW38NbGDPy8PbglLqLZPLERQSy/K46H5sW0eG+OtewzlfUj3gwC+WfPtZjfXc0b2YfMwgqO5Jc5uyqaCyk7V0taTikT3bB/EnTA0ezgZHEFX6Wc4ubJAwn0a/mO/rYOV44bHIyPl8GtR6K1xtzhYQCsPaCb1bT/cccJO6257KABzX29s/kYAvzsItuPoqm7KnL1mzg7qk9PX8YOCGRtai73z4pydnU0F+GOE3Zas/kVjoj8TUS+aSK9h4gsFZHhVmm/EpF9IqKvtDqJ4vJqPtlxgqvH9aNfoH1uSnOXmzg7at6IMFJOlpBbUuXsqmguIiE9n97+3hzOc8+mVpt+0YvIEODnmOc0a2wi8DRgPbXpm0AIcLst66E5zwfbsqisMXLPxUOcXRW3d+lIc7PaujTdrKbBzswiUk6WcLqs2m2XIrf1lcWvgBSlVFIT+8YD54ADdQlKqUrgfeBhG9dDc4KtRwtYlniU2IggYvq435BNVzMkJIB+PX15Z1OGW365aLb1xe7s+v+761LkrQo4IhIlIjUi8kyj9GWWlTYniogPcAvwURPHpwEvAz5AjYgoEfncsvsTYISITOvQK9GcKjmrmFvf2UFljZF9J0v0F6QN7Dp+hryz58gsrGDR2+75i1aznYpqI0CrbyNwRa0KOEqpI8A/gV+JSDCAiDwF3An8xHJFEwcEApuaKOI2IAPzDM1TLdtDln17MM8CfVn7X4bmbNsyCutnhDaa3PPXl6vZllFYfx/OuVoTa1NznVwjzZn2ZZcwun+PNt1G4Gra0qT2DOABPC4id2Huj7lVKbXesj8O80zMe5s4NgUIBzYopbZZtiwApZTJsj+una9BcwF1dz0L7vvry9XU3RhbNxvD1ymnKCzrnPcdaS07VlDOkfwyrp0Q7tYDZlo9LFoplSMirwK/sRz3oFLKelnMfkCpUqq6icNHAt6YF0JrymkgurV10VzP2apaAG6Ji2DB+P5u+4FwJdZDwHt28+KP3xzgp+/t5KO74wjw0Xc0dCXrLfdjzbHcn+Wu2vquPYy5H2azUuq1Rvt8MQ8KaMoEzFc/e5rZX4l5GWjNTW06XECgnxdLrx6pZzi2odiIoPrg3S/Ql7vfT2bJ+0n866eT8PHUC7R1FevS8hjWpzsDevk5uyod0uomNRG5BPMw5q3AdBFpPAtzIeY+nKaMB44qpRovG12nF6BXnHJTSik2Hy5g+pDeOtjY0exhYbx0/Rh+PFrIrz7Zg9Gk51nrCorKq0nKLGLeCPe+uoHWj1KbAKzCPHAgHjgOPN8oWzrgLSJNTds7Aqvh0E0YBBxsTV0013Mkv4zc0ipmDO3t7Kp0etdOCOf3Vw7nu/25/PzDZF5LOKxHr3VyG9LzMSmYO6KPs6vSYRcMOCISBXwHrAV+Yemj+QNwhYhcbJV1o+XfyU0UcwYYKyKXikhc3Ug3S/mBmPtvNjZxnOYGNh02X5xepAOOQ9w1YzA/Gd+fdQfyeGnNIW58cysf7zhOda15NdTkrGJeSziiA1Ensf5AHn16+DKqfw9nV6XDWuzDEZE+mANNGrDYMqIMzDdrPgq8AEwDUEplisgOYD7m5aGtPQW8A3yBua9nBrDZsu9KoBrzFZTmhjYdPs2g3v6EB7l3+7I7iQr1RzB3jNaaFL9duY8/fJ3KkBB/DuaWYTQpvDwM/P6q4ZZ1hwSDwOG8s+w/Vcr4AUGMDu+Jl4fgYRC8PAwcyCklNbuEqUN6Nxj0kZxVzDdHq+k+qFgPBnGwqhojGw+f5toJ/d1uOemmtBhwlFK5wHlr/iqljMDw849gGfBXEblfKVVhlX8/MKWZp7kF+LTxss8i0gtzkJqHuX/nt0qppm4qFcyB7y5L0j+Bx5XlBgYRGWcpZzjmwPkzpVRzgxe0NjpXa2RbRhE3TOw8C6C5g7jBvfHxOkJNrQlPDwO/vCSKgrIa/rvvVP39UNVGE099mdrk8e9vzWqh9ENEhfgzOCQAg0j9ctffZG5z2/s/3NWPRwuoqDZ2iuY0sP1s0R8CjwH3YZ5ZoEWWYDAb87Dpxl7DfOUTBowDvhWRFMuy0daWYF6KeizmH3zrgGPAGyLiDXwJvAq8DtwDfCkiQ5sZvq210a6sM1TWGLkoSjenOVJzs2ZfOaYvi9/eRrXRhKfBwO+uHEZUSHeMSvHVnmxW7spGYV5L6IrRfZkVE4rRpFiflse6A3nUDUOoNSmyCivIKiqvD2B106nogOM46w7kE+DjSdxg91yOoDGbBhylVK2I/BTzMOjW6APcYZnJoJ6I+APXAaOUUmXAZhH5CrgVeLxRGbcDf1FKnbQc+xfgbuANzAMcPIFXLVc8fxORhzEHudXteIlaI5uPnMbDIEwdom/0dDTrIdPWacvvbnr5hgAfT77dl0NNrQkvTwM/nT6ofv+Q0AA2Hj5dv+8vC8cRGxFEclYxN7+9jepaEwYRfUOvA5ksPwRmRod0niHwSimX2zAPo65olPYw8HUTeUuAKVaPJwJnLf//NfBdo/zfAL9popwlQBKQ5OfnpzBfLentAlufW19RYYv/7PR66K11m3e/YapH3A3Ku9+wVu/z7jdM9b//A9X3ztecXv+utHn3jVYRj32j/EfEO70ubdiSWvpud9V1aAKAxvfslABNTUEcYNlnnS/A0rfTeF+z5Sil3lJKTVRKTQwPD3d60HWFLSEhocX9RWXn8O0fzeN3LHB6XZ15HtxpO5edRsnWFZzLTmv1vnPZadwSG4Z3SASHckud/hq6ynvh2X99iYdBOLlzjdNfd2vPwYW4asApAxqPAeyBeZLPC+XtAZQp86tvSzlaG/14tBClYMbQEGdXRbOzaf088TQInyWfdHZVuox1B/KYHNnrgsu0uxNXDTiHAE8RGWqVNhZoashNqmVfU/lSgTHScDzhmGbK0dpo0+HTdPf1ZGx4T2dXRbOzHj7CrGGhrNydTa3RdOEDtA7JKiznUF4ZczrB7ALWXDLgKKXKMd/L84yI+IvIdOAa4IMmsr8PPCQi/UWkH+bJRd+z7EsEjMCDIuIjIg9Y0jfYs/5dgVKKTYcLmDYkGE8Pl3wbaTZ2Q2w4p8+e44dDp51dlU5vnWWyzs4wnY01V/6muA/zhJ75wMfAvUqpVBGZISLWC3q/iXmdnX3AfuBbSxrKPPR5Aeb1eM5gXr9ngdJDojvsWEE52WcquUg3p3UZs4aFEuzvzadJulnN3tZ3ksk6G3PZOc6VUkWYg0Xj9E2YBwPUPVaYZz14tJlydgOxdqpml7X5iHk6mxn6/psuw8vDwILx/Xl/ayZF5dX08u88fQuu5ExFNTszi7l35hBnV8XmXPkKR3NhGw8VMKBXNyKCO9cvMK1lN0wMp8ao+GJ3trOr0mm9u/kYRpNiQK/Ot2KLDjham9UYzXecXxQV0inmd9Jab1ifHozu31OPVrOT5Kxi/pFgvg/+6a9SO90ErDrgaG2WcuIMZedquVjPDt0l3TAx3DzR56nGt7hpHbX58Gnqljmqm0qoM9EBR2uzjYcLMAhMG6IDTld09dh+eHsY9OABO/D1Mk9hYxDw8jR0uqmEdMDR2mzz4dOMDg+kp5+Xs6uiOUGgnzdzR4Tx5Z7s+jV4NNvIOF2On5cHv5oT3Sln5tYBR2uTjYdOs/v4GYaG+ju7KpoTXT8xnOKKGr5Py3N2VTqNulm754wI48FLhna6YAM64GhtkJxVzM/+vRMFfLUnp9N1aGqtd/HQEMJ6+PBpJxs80NJqqXUL0dnrfb/reDGF5dXMG9m5bva05rL34WiuZ1tGIbVGc4+m0aTXRunKPAzCtRPCeSPxKC+uTueS4WFu/15Izipm8dvbOFdrwtND+OUlQxnU23zLX8bpMv624TC1RvstRLdmfy7eHgbiY0JtWq4r0QFHa7W4wcHUrWvcGTs0tbYZ0bc7CliWeJR3txxz+z6HbRkFVFn6pGqMipfXHmoynz0WolNKsfZAHtOjggnw6bxfy533lWk2N6JvDwzAxEG9ePSyYW795aJ13PGiSsC8CEpnWA20tLIWMP+m8vY08MK1YxjRrwcicOBUKY98lkKNUSF2WIjuYN5ZjhdVcG9855tdwJoOOFqr7TlxBqOCe2YOdusvFs024gYH42EQjCaFl4d7X/FmFZbzwbYsRvfvwaUj+zB1SO8G7/HoMPO8Zg99uI3jZ02Yw6ztrE3NQwQuGd55m9NADxrQ2iA5qwiACQN1sNHMy1k/edVwAH45x31HVRlNioc/TcFDhDdvncgDs5t+LbERQTw+xZfwXt345Sd7OFtVY7M6rEnNZcLAIEK7+9qsTFekA47Wajszi4kOC+hUC0JpHbNocgT+3h6cLK50dlXa7Z3NGezMLGbp1SPpF9jy/GXdPIVXbxxPTkkVT39pm2W1ThZXkHqqlEs78ei0OjrgaK1iNCl2ZRUzMbKXs6uiuRBvTwPTonqTePB0q5YYdjUHc8/y8ppDzBsRxrUT+rfqmNiIIH4xO4qVu7P5ck/HJzGtW/tm7og+HS7L1emAo7XKwdyznD1Xy6RI92w20ewnPiaE7DOVHD1dduHMLqS61sRDK/bQ3deTP107uk0T0T4wK4oJAwP5/Rf7OVlc0aF6rE3NIzosgEG9O//N1DrgaK2SZOm/mRihr3C0huruG0k86F4rgf5jw2FST5Xy3E9G0zvAp03HenoYePXG8SgFD/0nBaOpfVd3xeXV7MgsYl4XuLoBHXC0VkrKLKZPD1/CgzrfGh1ax/QP7MbQ0AC3Wnr6PzuP8/eEI8yM7s1lo9r3ZT8w2I9nrhnJjswi7vjXjnbNQPB9ej5Gk+rUswtYc7mAIyK9RGSViJSLSJaILGoh7yMisl9EzorIMRF5pNH+TBGpFJEyy7bW/q+gc0rKLCI2Mkivf6M1aWZ0CNsziqiornV2VS5ox7FCHv98H0rBtoyiDk1VE9HLD4PApsMF3PjmVra3cTmBtam59O3py+j+PdtdB3ficgEHeA2oBsKAxcAyERnZTF4BbgOCgMuAB0TkpkZ55iulAizbPHtVujPLPlPJqZIqJrnpsFfN/uJjQqk2mth61PXXb/l818n6u2hqjR1bc2bbsaL6/9eaFPd/tIu9J8+06tjKaiMbD59m3oiwLvNDzqUCjoj4A9cBTyqlypRSm4GvgFubyq+UelEptUspVauUOgh8CUx3XI27hqRMS/+NHqGmNWPSoCD8vD3coh+nrMp8FeZhgzVn4gYH4+1pMJflYb4J9iev/8iLq9OpqjG2eOzGw6epqjExb2TX6L8BEFcayigi44EtSik/q7SHgZlKqfkXOFaAXcCbSqk3LGmZQDfMgXU38IhSKqWZ45cASwBCQkJiV6xY0fEX1IIjxUbSi4wM6+VBVJCHXZ+rvcrKyggICOD91HP8eKqW1y7xw8PQNX6JWas7D11Za87Bq8lVZJeZePHibi77i10pxSMbKwn0EcaFeLT589fUebD+LPcNMPBJejWbsmvp5y9cGunF2WrV5PO8vfccu/Nr+dtsPzzd6HPV0nth1qxZyUqpic0d62pT2wQApY3SSoDurTh2KebA8i+rtMWYg5AAvwTWiMgwpdR517xKqbeAtwBiYmJUfHx8W+vepI0HT/Pt/lP07dmNAB9P8s+eIz2nlM1HCjAp8PY08vHdrjnpYWJiIvHx8bywZyOTBvtwyewpzq6SU9Sdh66sNefghG8WT36xn4hRkxgc4poBOi2nlII1m3joslEsmjKwzcc3dR7iG+W5ci4kHsznNytS+FdqNQAeUsv8sf0YHd6TkO4+9A7wJiUhiciQHgQNGeWSn//mdOTz4NCAIyKJwMxmdm8BfgH0aJTeAzh7gXIfwNyXM0Mpda4uXSm1xSrb8yJyOzAD+LptNW+f/+7N4b6PdjVI8/E04OtlqF+3vLrWxH92HnfZN1xJZQ0H885yxei+zq6K5uLio0MA8/BoVw046y03Wc6x85xl8TGh3BI3kL99fwQFGJXiq5Rsvmh0o2haTimL/2mf5Q5ckUP7cJRS8UopaWa7CDgEeIrIUKvDxgLNziEhIncCjwOXKKUutBqUwny1Y3dF5dU88cW++scGgQdnR5H+x8t4947J+HqZ230FWLkrm+/25TiiWm2263gxSsFEfcOndgEDevkxOMSfRBceHr0uLY9xAwIJ7WH/Ocsujg7Fx/I59/Uy8Ok9U9nz1FzW/fpibpwYXrfSR/1M212BSw0aUEqVAyuBZ0TEX0SmA9cAHzSVX0QWA38C5iqlMhrtGygi00XEW0R8LUOme2O+krKrymojd763k7JztXh7mN9w3p4GZsaEIiLERgSx/K44HpoXw79/OpmxAwK5/6NdfLzjuL2r1mZJmUV4GIRxAwKdXRXNDcRHh7Ito5DK6pY7zJ0ht6SKvSdLmDvCMfe8WH/Ol98VR2xkLwL9vBka1p2FkwbWB6OutLaUq/XhANwHvAvkA4XAvUqpVAARmQF8p5Squ15/FggGdlp1Un6olPo55n6fZcAQoArYA1yulLLrT4lao4lffLyblJNnWLY4lpDuPmzLKCRucHCDS+bYiKD6xxMHBXHf8l38duU+iiuquXfmEJfpdN2ZWcyofj3w83bFt4rmauJjQnh3yzG2HStkloutXLkuzdycNs9BAQcafs4bpy+/K67J74bOzOW+RZRSRcCCZvZtwjywoO7xoBbKSQXG2LyCLVBK8eSXqaxPy+OZa0bW38F8oTeTn7cnb982kYc/TeHF1QdJzyklpk934gb3duobsdakSDlxhlviIpxWB829TB7UC18vAz8cPO16AedAHhHBfkSFukb/UnPBqDNzuYDjzv6x4Qgf7zjOffFDuG1qZJuO9fIw8H8Lx1Fda+KrlBwkJQcfryNO7UzMLDVxrtakJ+zUWs3Xy4Opg4NJPJgPNHe/tuOdraph69ECbp8a6TKtB12RS/XhuKvkrGLu/TCZv6w7xLXj+/PIpTHtKsdgEEb1Nw/Sc4XOxMPF5vXdY/WEnVobxMeEkllYQWZBubOrUm/joQJqjMph/Tda0/QVTgclZxVz81vbqDaaMAjcOGlAh35BxQ3ujY/nEc7VmuyydnpbHCo2Mqi3PyHd2zaTrta1xceYh0f/cOg0kS4y5f66A7kE+Xl1uSYsV6OvcDooIT2faqP5SkCApA5MBAjmdt2P7o4jopcfgd28nDY6TCnF4WKj/oBqbRYR7E/fnr78+8fMDk2MaSs1RhMb0vOZPSwMTw/9ledM+ux3gFKKnZnmJi+DDYc3xkYE8dsrhlFQXs2G9PwOl9ceR0+XU1aD7r/R2iw5q5j8s+fIKChn8dvbnB50dh4rorSqlrkjXGsQQ1ekA04TSs6pVn1IViSdYPuxYm6bGsFv6sba2+iKYM7wMMJ6+PDhtiyblNdWyVl6wk6tfbZlFNYvN33OBW5qXJeWh7engRlDQ5xaD00HnCYVn1MsusAvs4zTZSz96gDThgSzdP5I7p8VZdPmJ08PAzdNGsjGw6c5XtixJWzbY/X+XLwNcKa82uHPrbm3uhmU6zjzKlkpxboDeVwU1Rt/H91l7Ww64DTjXK2JL3ZnN7mvxmjiV//Zg7engVcWjsNgp5leb548EIMIy3c49ionObOIxIOnqTbB4ne2O71JRHMvdTc1LhjXDwUUlDnvR0t67llOFlfq0WkuQgecZgjw8c7jrNh54rx9r64/xN6TJbxw7Wj69LTfnEx9evoyZ3gonyad5Fyt46YK+XpvTv0CVc4emq25p9iIIP6ycBwRwX68vSnjwgfYyTrLZJ2X2HmyTq11dMBpQpCP8O5PJxE3KJhHP9/LE6v2UV1rHom2PaOQ1xOPcuPEAVzugBmUb4mLoKi8mu/25dr9uerULRNsoGvN86TZlodB+NlFg9h9/Ex9n6Cjra+brLO7/Sfr1C5MB5wm9PQRZsWE8t5PJ3HPzMEs336cm9/exld7srn7/STCuvvw1PwRDqnL9CG9iQz2c+jggb0nSxjetzvXDvXqMtOma/ZxfWw4Pbt58c9Nxxz+3OtS89h7soQR/RqveKI5iw44LfD0MPDby4fz2qIJ7M8u4cFP9lBaVUtRRQ3puS0u0WMzBoOweEoESVnFpOc2XpvO9o4VlJOee5YbYgdw1RBvHWy0DvHz9mTxlIGsSc116OCX5Kxi7l2eDMDnySd1P6SL0AGnFa4c05dFk/+3OqDR6Nh+jetjw/H2NDjkKmdNqrnp7tJRXWeddc2+bp8WiYdBeHeL465ytmUUUmtZ5bDWwZ9XrXk64LTSVWP71S+a5uh+jSB/b64a05dVu7IpO1dr1+davT+XMeE96R/Yza7Po3UdYT18mT+2HyuSTlBSUeOQ56yboUPQ/ZCuRAecVjpvMSUHNzXdEhdBebWx2aHatpBTUsmeE2e4dKS+utFs666LBlNRbeQjBy0yWPfDbOHEcN0P6UL0nVBt4Mz1K8YPCGRE3x68vTGDkspqu6yVs2a/uTntct2cptnYiH49mB4VzHs/HuNnFw1qcGOoPWxIy6e7jyd/XDDa7s+ltZ7+S7gJEeHiob3JKqrgL2sPsfiftp+janVqLtFhAQwOcY0FqrTO5a4Zg8krPce3+07Z9XlMJsWGg/lcHBOig42Lcbm/hoj0EpFVIlIuIlkisqiFvEtFpEZEyqy2wVb7x4lIsohUWP4d55hXYR++3h4AmJTtb8gsLDvHjmNFXKab0zQ7mTk0hKjQAP66/jCvJRy228ixfdklnD57jkuG6Zs9XY3LBRzgNaAaCAMWA8tEpKWlA/+jlAqw2jIARMQb+BL4EAgC/g18aUl3SzOGhuDlYZ5Gx2Cw7Vo569PyMCk9Ok2zH4NBmDvcvDibva7SAb5Pz8cg5oXgNNfiUgFHRPyB64AnlVJlSqnNwFfAre0oLh5zH9WrSqlzSqm/YR60MttW9XU088CFKfTy96Z3gA9jwnvarOzV+3MZ0KsbI/rqm+Q0+/H1st9Vep0N6XlMGBhEL3+3/W3ZabnaoIFooFYpdcgqLQWY2cIx80WkCMgB/qGUWmZJHwnsVXXzpJvttaSvblyIiCwBlgCEhISQmJjY7hdhb7fFCK/uquIPH37PnAivDpdXUaPYeKiCuRGe/PDDD/XpZWVlLn0eHEWfB9udA/+zRgxiDjgGAZ8zWSQmnux4BS2Kq0zsz67k+mgvu/zN9HuhY+fA1QJOAND4dvoSoHsz+VcAbwF5wBTgcxE5o5T62FJWSWvLUkq9ZSmLmJgYFR8f3576O8RMpdhRsp1vs0p59Mbp9PDtWND5ck82RrWHuy+f3GDkW2JiIq58HhxFnwfbnYN4IHRQNg9+soebJkdw14JRHS7T2kfbjwP7WHLlVKLDmvvaaD/9XujYOXBok5qIJIqIambbDJQBjdt0egBNziOjlDqglDqllDIqpX4E/gpcb9ndprLciYjwuyuGc6ayhtcTjna4vNX7cwnt7sN4Jy1nrXUtV4/rz6TIILYcKaBhA0THfZ+WR3hQN4aG6pGWrsihAUcpFa+Ukma2i4BDgKeIDLU6bCyQ2tqnwNxPg+WYMSJivVjNmDaU5dJG9e/JT8b3590txzhZ3P45qiqrjSQePM2lI/vYbV0fTWts4cQBZBSUszPTdoMGKquNbD5SwJzhYTT82GuuwqUGDSilyoGVwDMi4i8i04FrgA+ayi8i14hIkJhNBh7EPDINIBEwAg+KiI+IPGBJ32DXF+FAD8+LQYCX1xxsdxkbD5+mssbIZXp0muZAV47pS4CPJ5/stN3MA1szCjhXa2K2Hg7tslwq4FjcB3QD8oGPgXuVUqkAIjJDRMqs8t4EHMHcTPY+8Gel1L8BlFLVwALgNuAMcCewwJLeKfQL7MZdMwbxxZ5T7D15pl1lrN6fS6CfF5MH9bJx7TSteX7enlw9rh//3ZdDaZVt5ldbn5aPv7cHUwbr97KrcrmAo5QqUkotUEr5K6UGKqU+stq3SSkVYPX4ZqVUsOX+m2GWoc/WZe1WSsUqpboppSYopXY78rU4ws9nDiHY35vnvk1rc3v49oxCvt2Xw4QBgXh5uNxbQevkbpw4gKoaE1/t6fjMA0opNqTlM2NoCD6eHjaonWYP+lvGzXX39eJXc6PZfqyIh1aktPpGuuSsYm55ZzvVtSY2HSnQ64VoDjcmvCfD+nTnP00s495WqadKyS2tYrZeStql6YDTCcSEBSDAqt3Z3PjmVn48UtBi/gOnSvnNij3UGM1XRCaT0uuFaA4nItw4aQD7sktIPdX4Doa22ZCejwjM0rMLuDQdcDqBnZnF1A3KqTUpfvreTl5cnU5+aVWDfDkllTz8aQpX/n0Tp8+ew9MgTlnfR9Pq/GR8f7w9Dazo4FXO9+n5jA0PJKS7j41qptmDq934qbVD3OBgvD0N1NSa8PAwMG5AIMt+OMrbmzK4emx/JkYG8eWebHZlFQPC3TMGc398FEdOl7Eto5C4wcF6vRDNKQL9vLlsZB9W7c7mt1cMr5/6pi3yz1aRcuIMv5kbbYcaarakA04nULc4nHXwyCos593Nx/h4xwk+32WeOsQg8Nqi8Vw+um/9cTrQaM5246QBfJVyijWpuVwzrn+bj09MPw3AJcPDbF01zcZ0wOkkGgePiGB/foAA3AAACwVJREFU/nDNKHp08+IfG47U3xGbUVDutDpqWlOmDg5mQK9ufLLjRLsCzmfJJ+ju40lltX2XX9c6TvfhdHLxMaH4eBl0X43msgwGYWHsALZmFJJV2LYfRFuPFrAjs5iz52pZ/M52PdrSxemA08nVNbc9NC9Gr+2uuazrJ4YjwGOf721T0Hh57f8mlrfXcgea7eiA0wXERgRx/6woHWw0l3XqTBUisC2jqNULs320/TjJWcV46NGWbkP34Wia5nTbMgqpmyijqsbEpsOnW/yB9OPRAp76cj8zo0O4f9YQdmYW69GWbkAHHE3TnC5ucDA+XgbO1ZhQwDd7c7g1LoLggPPvqzlWUM69H+4isrc/f180nh6+XkwepK9s3IFuUtM0zenq+hofvjSGxy6L4URRBde/sZXjhQ2X3iiprOFn/96JQeCd2yd2ePFBzbF0wNE0zSXU9TXeGx/FR3dPobiimmuXbWHfSfO0N7VGEw98tIsTRRW8cUssEcH+Tq6x1lY64Gia5nJiI3rx2c+n4ePpwU1vbeWdzRksfHMrmw4X8NyC0UzRgwPckg44mqa5pKjQAFbeN43eAT788Zs0dh0/g4dBGKKXj3ZbOuBomuaywnr4cs34fv9LUHpmc3emA46maS5tZnQovnq2jE7B5QKOiPQSkVUiUi4iWSKyqIW834lImdVWLSL7rPZnikil1f61jnkVmqbZip4to/NwxftwXgOqgTBgHPCtiKQopVIbZ1RKXW79WEQSgQ2Nss1XSq23U101TXMAPbN55+BSVzgi4g9cBzyplCpTSm0GvgJubcWxkcAM4H171lHTNE1rH5cKOEA0UKuUOmSVlgKMbMWxtwGblFKZjdKXi8hpEVkrImNtVE9N0zStjVytSS0AKG2UVgJ0b8WxtwHPNkpbDOzCvBTML4E1IjJMKXWm8cEisgRYAhASEkJiYmLbat4JlZWV6fOAPg+gz0EdfR46eA6UUg7bgERANbNtBsYDFY2O+Q3w9QXKvQgoAwIukC8dc59Oi/WMjo5WmlIJCQnOroJL0OdBn4M6+jy0fA6AJNXCd6tDr3CUUvEt7bf04XiKyFCl1GFL8ljgvAEDjdwOrFRKlV2oCpivdjRN0zQHc6k+HKVUObASeEZE/EVkOnAN8EFzx4hIN2Ah8F6j9IEiMl1EvEXEV0QeAXoDW/6/vfsNkeuqwzj+fezWpGYTQ9IaocUsqU1ttqQmVgQ1GGgkti9EzZuyUQzYBlsDYt+0oJKthhZB8IW01eCW1kRSpY0RQXwhNGIKkQZNrdumKREarVFpzJ/tpln/8PPFOWPGcXebnc2cO7P3+cCFvfdMht99MtzfzN2zZzp2AmZmNqWuajjZ3cAVwN+APcBdkadES1onqfVTzCeA08DTLccXAo8Ap4BXgY8Bt0aE/0zZzKwC3TZpgIj4O6mJTDb2K9LEguZje0iNqfWxo8DqTtRoZmYz142fcMzMbA5ywzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Mi3HDMzKwINxwzMyvCDcfMzIpwwzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzIrqu4UjaJumQpAlJj13E478k6S+Szkp6VNK8prEBSU9LOifpiKQNHS3ezMym1HUNB/gzsAN49M0eKGkjcB9wC7AcWAHc3/SQPcBvgaXAl4EnJV11qQs2M7M313UNJyL2RsQ+4ORFPPyzwEhEjEbEKeDrwBYASSuBtcD2iHgjIp4Cngc2daZyMzObTl/VBczSIPCTpv3ngGWSluaxP0TEWMv44GRPJGkrsDXvTkj6fQfq7TVXAq9VXUQXcA7OoME5TJ/B8un+Ya83nH7gTNN+4+eFk4w1xq+e7IkiYiewE0DSoYi4+dKW2nucQ+IcnEGDc5hdBkVvqUnaLymm2A608ZSvA4ua9hs/j00y1hgfw8zMiivacCJifURoiu3DbTzlKHBT0/5NwF8j4mQeWyFpYcv4aPtnYGZm7eq6SQOS+iTNBy4DLpM0X9JUt/6+D3xO0ipJi4GvAI8BRMRR4DCwPT/HJ4HVwFMXUcbO2Z7HHOEcEufgDBqcwywyUERcykJmTdIwsL3l8P0RMSzpXcALwKqIOJ4ffw9wL3AFqZl8PiIm8tgAqQF9ADgOfCEiftH5szAzs1Zd13DMzGxu6rpbamZmNje54ZiZWRFuOE0kLZH0Y0njkl6RNFR1TSVMt36dpFvyOnTn8rp00/5hV6+SNE/SSP5/H5N0WNKtTeN1yWG3pBN5bcKjku5oGqtFBs0kXSfpvKTdTceG8utkXNI+SUuqrLGT8p+ynJf0et5eahqbcQ5uOP/rIeAfwDJgM/CIpElXJphjJl2/TtKVwF7gq8AS4BDww+LVldEH/BH4CPB20ozHH+UFYOuUw4PAQEQsAj4O7JD0vppl0Owh4NnGTr4efBf4DOk6cQ54uJrSitkWEf15ux7az8GTBjJJC4BTwI15SjWSdgGvRsR9lRZXiKQdwDURsSXvbwW2RMQH8/4C0pIWayLiSGWFFiLpd6TFYJdSwxwkXQ/sB74ILKZmGUi6HfgUaWbsuyPi05IeIDXkofyYa4EXgaUty2jNCZL2A7sj4nstx9vKwZ9wLlgJ/KvRbLIp116riUFSBgBExDhwjBpkImkZ6TUxSs1ykPSwpHPAEeAE8DPql8Ei4GvAPS1DrTkcI90VWVmuuuIelPSapGckrc/H2srBDeeCfuBsy7EzpHXZ6mqq9ejmdCaSLgd+ADye373XKoeIuJt0butIt9EmqFkGpJXnRyLiTy3H65bDvaSvfbma9AefP82fZtrKwQ3nAq+99v9ql4mktwC7SO/WtuXDtcshIv4dEQeAa4C7qFEGkt4LbAC+NclwbXIAiIhfR8RYRExExOPAM8BttJlDr68WfSkdBfokXRcRL+djdV97bZT0nUPAf+/bX8sczUSSgBHSL0Fvi4h/5qFa5dCijwvnWpcM1gMDwPH0kqCftMzWKuDnNK3fKGkFMI90/aiDAETLOpYXnUNEeMsb8ATpW0IXAB8ifUQcrLquAufdB8wnzVDalX/uA67KGWzKx74BHKy63g7m8B3gINDfcrwWOQDvAG4nX2CBjcA4abZaLTLIObwNeGfT9k3gyZzBIOnW+7p8ndgNPFF1zR3KYXF+DTSuB5vz62FluzlUflLdtJGme+7LoR4HhqquqdB5D+d3Ls3bcB7bQPrl8RukGUsDVdfboQyW5/M+T7pd0Ng21yWHfEH9JXA6X0yeB+5sGp/zGUyRyzBpplZjfyhfH8ZJXwC5pOoaO/h6eJZ0m+x0fjP20dnk4GnRZmZWhCcNmJlZEW44ZmZWhBuOmZkV4YZjZmZFuOGYmVkRbjhmZlaEG46ZmRXhhmPWIyQtkjQs6YaqazFrhxuOWe+4GdgOXF51IWbtcMMx6x1rSF8V8ELVhZi1w0vbmPUASS8C72k5vDciNlVRj1k73HDMeoCk95NWMx8FHsiHT0TEK9VVZTYz/j4cs97wHOnL0L4dEQerLsasHf4djllvGATeCvym6kLM2uWGY9Yb1pK+r+dw1YWYtcsNx6w3rAGORcTZqgsxa5cbjllvWIWnQ1uP86QBs95wGlgraSNwBng5Ik5WXJPZjHhatFkPkHQjMAKsBuYD6yLiQLVVmc2MG46ZmRXh3+GYmVkRbjhmZlaEG46ZmRXhhmNmZkW44ZiZWRFuOGZmVoQbjpmZFeGGY2ZmRfwH/x/QFC6b8qsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "y_pred = model.predict(X_valid)\n", "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "BaS5Yqbx5jk7" }, "source": [ "## Using a Simple RNN" ] }, { "cell_type": "markdown", "source": [ "Let’s see if we can beat that with a simple RNN. It just contains a single layer, with a single neuron. We do not need to specify the length of the input sequences (unlike in the previous model), **since a recurrent neural network can process any number of time steps** (this is why we set the first input dimension to None). By default, the SimpleRNN layer **uses the hyperbolic tangent activation function**. It works exactly as we saw earlier: the initial state $h_{init}$ is set to 0, and it is passed to a single recurrent neuron, along with the value of the first time step, $x_0$. The neuron computes a weighted sum of these values and applies the hyperbolic tangent activation function to the result, and this gives the first output, $y_0$. In a simple RNN, this output is also the new state $h_0$. This new state is passed to the same recurrent neuron along with the next input value, $x_1$, and the process is repeated until the last time step. Then the layer just outputs the last value, $y_{49}$. All of this is performed simultaneously for every time series." ], "metadata": { "id": "4H7cLxnfAVEW" } }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "LlXfar6n5jk7" }, "outputs": [], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.SimpleRNN(1, input_shape=[None, 1])\n", "])" ] }, { "cell_type": "markdown", "source": [ "Note that for each neuron, a linear model has one parameter per input and per time step, plus a bias term (in the simple linear model we used, that’s a total of 51 parameters). In contrast, for each recurrent neuron in a simple\n", "RNN, **there is just one parameter per input and per hidden state dimension** (in a simple RNN, that’s just the number of recurrent neurons in the layer), plus a bias term." ], "metadata": { "id": "h11FMMnGCCeg" } }, { "cell_type": "code", "source": [ "model.summary()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "E082xqnTBttm", "outputId": "47389102-2276-41ab-c5fd-173c8a079662" }, "execution_count": 14, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_1\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " simple_rnn (SimpleRNN) (None, 1) 3 \n", " \n", "=================================================================\n", "Total params: 3\n", "Trainable params: 3\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ] }, { "cell_type": "code", "source": [ "optimizer = keras.optimizers.Adam(learning_rate=0.005)\n", "model.compile(loss=\"mse\", optimizer=optimizer)\n", "history = model.fit(X_train, y_train, epochs=20,\n", " validation_data=(X_valid, y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DFkksB7EBsjd", "outputId": "4e89dcd8-f38c-4e73-86eb-5bb18bf7ca58" }, "execution_count": 15, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 13s 54ms/step - loss: 0.0967 - val_loss: 0.0489\n", "Epoch 2/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0369 - val_loss: 0.0296\n", "Epoch 3/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0253 - val_loss: 0.0218\n", "Epoch 4/20\n", "219/219 [==============================] - 18s 80ms/step - loss: 0.0198 - val_loss: 0.0177\n", "Epoch 5/20\n", "219/219 [==============================] - 15s 68ms/step - loss: 0.0166 - val_loss: 0.0151\n", "Epoch 6/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0146 - val_loss: 0.0134\n", "Epoch 7/20\n", "219/219 [==============================] - 11s 51ms/step - loss: 0.0132 - val_loss: 0.0123\n", "Epoch 8/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0124 - val_loss: 0.0116\n", "Epoch 9/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0118 - val_loss: 0.0112\n", "Epoch 10/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0116 - val_loss: 0.0110\n", "Epoch 11/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 12/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 13/20\n", "219/219 [==============================] - 12s 56ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 14/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 15/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 16/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 17/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 18/20\n", "219/219 [==============================] - 12s 53ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 19/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0114 - val_loss: 0.0109\n", "Epoch 20/20\n", "219/219 [==============================] - 11s 52ms/step - loss: 0.0114 - val_loss: 0.0109\n" ] } ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "xp6oMcGi5jk8", "outputId": "345df898-d649-4d5b-88f4-465dcea37f37", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 0s 8ms/step - loss: 0.0109\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "0.010881561785936356" ] }, "metadata": {}, "execution_count": 16 } ], "source": [ "model.evaluate(X_valid, y_valid)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "id": "ArbK1I4Z5jk8", "outputId": "54db10c9-856c-4d19-f5d0-dcc062d70344", "colab": { "base_uri": "https://localhost:8080/", "height": 291 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUdf7H8dcnBRISAglI70iR3jECAkexoih4h6DCiWI5VA4RuyJwKgpnOVGagAgi+lMUK9IiVSmK0gRBQOnSEwhp+/n9MZuQxAXStoR8no/HPLI7892Z92w2+8m074iqYowxxhS0IH8HMMYYc3GyAmOMMcYrrMAYY4zxCiswxhhjvMIKjDHGGK+wAmOMMcYrrMAYY4zxCp8WGBGJEZG5InJKRHaLSN9ztBMRGSMiR9zDGBGRTNPVPY8E9zDFd2thjDEmJ0J8vLzxQDJQHmgGfCEiP6nqpmztBgE9gaaAAguAncCETG2aqup270c2xhiTFz7bghGRCKAX8LSqJqjqcmAecLuH5v2Bcaq6R1X3AuOAAb7KaowxJv98uQVTF0hV1W2Zxv0EdPTQtqF7WuZ2DbO1WSoiQcBKYKiq7vK0UBEZhLNFRHh4eMuqVasCcOZMML//XoLKlROJiEjN8Uq4XC6CggLj0FUgZYHAymNZPAukLBBYeSyLZ9u2bTusqpfk6cWq6pMB6AAcyDbubiDOQ9s0oH6m53VwdpWJ+/mVQDGgNPAGsBEIuVCGli1barpt21RBdcYMzZUlS5bk7gVeFEhZVAMrj2XxLJCyqAZWHsviGbBW8/i978sSmQBEZRsXBcTnoG0UkOBeWVR1qaomq+px4CGgJnBZbsLExDg/jx3LzauMMcbklC8LzDYgRETqZBrXFMh+gB/3uKY5aJdOATnPdMeBA7BqFQClSzujjh694KuMMcbkgc8KjKqeAj4GRopIhIi0A24E3vXQfAYwVEQqi0gl4GFgOoCINBSRZiISLCKROCcA7AW2XDDE3r3QpQusWkVwMERF2RaMMcZ4i6+PIt0PhAOHgNnAfaq6SUQ6iEhCpnYTgc+ADTjHV75wjwPnFOc5wEngN6AGcL2qpuQoQXIyxMUBzm4yKzDGGOMdPr0ORlWP4lzfkn38MiAy03MFhruH7G0XA/XyHKJYMejUCYDoaNtFZowx3uLrCy39KyQEFi2C2FjAKTC2BWMKk5MnT3Lo0CFSUnK2wZ5dqVKl2LLlwnuTfSWQ8hTVLBEREVSpUsUrp0UXrQKTmgoNGmQ8jYmBTec7dcCYAHLy5EkOHjxI5cqVCQ8PJ1PvSTkWHx9PyZIlvZAubwIpT1HM4nK52Lt3L4cPH6ZcuXIFPv/AuJLHl9aty3hou8hMYXLo0CEqV65MiRIl8lRcjMkuKCiI8uXLc+LECe/M3ytzDWRr1mQ8TN9F5lxdY0xgS0lJITw83N8xzEUmNDSU1NSc92aSG0WrwBQvDqtXZzyNiXFOKjt92o+ZjMkF23IxBc2bn6miVWAiIrIUmOho56cd6DfGmIJXtApMiRKwZw/s3w9YdzHGGONNRavAREQ4P93HYdK3YOxAvzGFy4ABA7j++utz9ZpOnToxePBgLyU6a8SIETRq1MjryykMitZpyiVKQHCws5vshhtsF5kxXnah/ft9+/Zl1qxZuZ7va6+9lt7Teo59/PHHhIaG5npZJu+KVoEJCoLGjTOOw9guMmO8a797dzTA559/zt13351lXPazl1JSUnJUBEqVKpXrLDHpf/DGZ4rWLjKA1q2dXWSqtovMFEmrVsELL2R0LO5VFSpUyBhKu7swT39+5swZqlatyuzZs/nb3/5GeHg4EydO5MiRI9x6661UqVKF8PBwGjZsyLRp07LMN/susk6dOnH//ffzxBNPULZsWcqVK8ewYcNwuVxZ2mTeRVajRg1Gjx7NPffcQ1RUFPXr1+fll1/Ospxt27bRsWNHwsLCqFevHl9++SWRkZFMnz49x++By+Vi1KhRVK1aleLFi9O4cWM+/fTTLG1GjhxJ9erVKV68OBUqVGDQoEEZ05YuXcrll19OZGQkpUqVok2bNmzcuDHHy/enorUFA9CmDUyeDNu3U7J2HYKCbAvGFE5DhsD69bl7zbFj4WzcCC6Xs0HfpAnkZmOgWTN49dXcLfNCHn/8ccaOHcvbb79NaGgoZ86coUWLFjz66KNERUWxcOFC7rnnHqpVq0aXLl3OOZ9Zs2bx0EMPsXLlStavX0/fvn1p2bIlt9566zlf88orr/Dcc8/xyCOPMHfuXIYPH0779u2JjY3F5XJx0003UaFCBb777jsSExMZMmQISUlJuVq/1157jZdffpkJEybQqlUrZs6cyc0338y6deto1qwZH330EWPHjmX27Nk0btyYQ4cOEefukDc1NZUbb7yRgQMHMmvWLFJSUvjhhx8IDg7OVQZ/KZoFBmD1aoLq1LGr+U2RcuKEkP5PvcsFJ07krsB4wwMPPEDv3r2zjHvkkUcyHg8aNIjFixcze/bs8xaYBg0aMHLkSADq1q3L5MmTWbRo0XkLTPfu3TO2au69914mTZrEokWLiI2NZcGCBWzdupVvvvmGypUrA05BateuXa7Wb+zYsQwbNoy+ffsCztbK0qVLGTt2LDNnzmT37t1UrFiR7t27ExoaSrVq1ahXz+nP9+TJkxw/fpwePXpQu3ZtAOrXr5+r5ftT0SswDRpAeLhzHKZfP+vw0hRaedmSWLjwDDfcEEFystOx+KxZGX2/+k2rVq2yPE9LS+PFF19kzpw57N27l6SkJJKTk+nk7gX9XJo0aZLleaVKlTh06FCeX/PLL79QqVKljOIC0Lp161x1Cnny5En27dv3l6LUvn17vvzySwBuueUWXnvtNWrWrMlVV13F1VdfTefOnSlZsiQxMTEMGDCAq666ii5dutClSxd69+5NtWrVcpzBn4reMZiQEGjZMsupylZgTFHRtq2LRYtg1KgsHYv7VUT65QNuY8eOZdy4cTzyyCMsWrSI9evX07NnT5KTk887n+wnB4hIlmMwBfWagpJ+hl3VqlXZunUrEydOJCoqiocffpgrr7ySU6dOATBt2jS+//57rrzySubNm0e9evWYP3++TzLmV9ErMODsJvvhB0hJISbGdpGZoiU2Fh5/PDCKiyfLly+nR48e3H777TRr1ozatWuzbds2n+eoX78++/btY9++fRnj1q5dm6sCFBUVRaVKlVixYkWW8cuXL6dBpp7dw8LCuO6663jllVdYs2YNW7ZsyfKapk2b8uijjxIXF0enTp1455138rFmvlP0dpGBU2D++1/YsIHo6Bbs2OHvQMaYdHXr1mXOnDksX76csmXL8r///Y+dO3fSvHlzn+bo1q0b9erVo3///owdO5bExESGDh1KSEhIrvrveuSRR3jmmWeoU6cOLVu2ZObMmSxbtowffvgBgOnTp5Oamkrbtm2JjIxkzpw5hIaGUqdOHXbu3MnEiRO54YYbqFy5Mr/99hs///wz9913n7dWu0AVzQLTurXzc/VqYmJa2C4yYwLIU089xc6dO7nmmmsIDw9nwIAB9OvXj82bN/s0R1BQEHPnzuWuu+6iTZs21KhRg3HjxnHzzTcTFhaW4/k8+OCDxMfHM3z4cA4ePEi9evX46KOPaNq0KQClS5dmzJgxDBs2jJSUFBo0aMDMmTOpWbMmBw8eZNu2bdxyyy0cPnyY8uXL069fPx599FFvrXbBUtUiM7Rs2VJVVdXlUi1TRvXOO/XJJ1WDglTT0jRHlixZkrOGPhBIWVQDK8/FmGXz5s35nsfJkycLIEnBCaQ8Ocmyfv16BXTt2rV+z1KQzvfZAtZqHr9zi+YWjIizm2z1aqIHOKdrxsf7/3RNY0xgmTt3LhEREdSpU4ddu3YxdOhQmjZtSosWLfwdrVAomgf5wSkwmzdTrkQCYGeSGWP+Kj4+nsGDB9OgQQP69evHZZddxvz58+2+PDlUNLdgwDkO43JR4+gPwJUcPQo1avg7lDEmkNxxxx3ccccd/o5RaBXdLRj3gf5Ke5yOL20LxhhjClbRLTDlykGNGpT5zSkwdi2MMcYUrKJbYABatyZys23BGGOMNxTtAtOmDSF7dnMJh6zAGGNMASvyBQbgipA1tovMGGMKWNEuMC1aQFAQHYqvti0YY4wpYEW7wERGQoMGtMEKjDGBbMSIETRq1Oiczz0ZPHjwBbv4z8uyvSX7XTovBkW7wAC0aUPjpDUcPaL+TmLMReeGG244503CtmzZQlRUFN98802u5zts2DC+/fbb/MbLYvfu3YgIa9eu9fqyigorMG3aUDr1CCUO7vR3EmMuOgMHDmTJkiXs2rXrL9PefvttqlWrRteuXXM938jISMqUKVMACQNrWRcbKzDuA/01Dq32cxBjfGTVKnjhBeenl1133XWUL1+eadOmZRmfkpLCu+++y2233YaqMnDgQGrWrEl4eDh16tThpZdeOu99V7LvtkpLS2PYsGFER0cTHR3NkCFDSEtLy/Kar7/+mg4dOhAdHU1MTAxXXXUVW7ZsyZjeuHFjwLlrpYhk7F7LviyXy8WoUaOoWrUqxYsXp3Hjxnz66acZ03ft2oWI8NFHH9GtWzdKlChBgwYNWLBgQa7eu6SkJIYMGUL58uUJCwvj8ssvZ/ny5VnewwcffJBKlSpRvHhxqlatymOPPZYx/eOPP6ZJkyaEh4cTExNDx44dOXjwYK4y5FfR7SomXaNGJAeHUffkGqCPv9MYk3NDhsD69bl6SfixY7Bxo9PDa1AQNGmSu15emzXL1b2aQ0JC6N+/P9OnT+fZZ5/NuN3wZ599xuHDh7nttttwuVxUrlyZDz74gEsuuYTVq1czaNAgypQpw8CBA3O0nHHjxjF58mQmT55MkyZNGD9+PLNmzcrSKeWpU6cYMmQITZo0ITExkdGjR9OjRw82b95MsWLFWLJkCZ07d+brr7+madOmFCtWzOOyXnvtNV5++WUmTJhAq1atmDlzJjfffDPr1q2jWbNmGe2efPJJXn75Zd58801Gjx5Nnz592L17N5GRkTlap+HDh/PBBx8wdepUatWqxX//+1+uvvpqfv31VypWrMjrr7/O3Llzef/996lRowZ79uxh69atABw4cIA+ffrwwgsv0KtXLxISEvjuu+9ytNyCZAUmNJSDFZvTbM9qUlOdOyobc7GSEyec4gLOzxMnvN6N+MCBAxkzZgwLFy6ke/fugLN7rHv37lSpUoXQ0FBGjhyZ0b5GjRr88MMPzJ49O8cF5tVXX2X48OH8/e9/B5wikP22wr169cryfNq0aURFRbF69Wrat29P2bJlAShTpgwVKlQ457LGjh3LsGHD6Nu3LwAjR45k6dKljB07lpkzZ2a0+/e//02PHj0AeP7555kxYwbr16+nffv2F1yfU6dO8dZbbzFlyhSuu+46ACZMmMDixYsZP348o0ePZvfu3dStW5cOHTogIlSrVo0rrrgCgH379pGSkkLv3r2pXr06gE9OVMjOp1+nIhIDvA10Bw4Dj6vqex7aCfAicJd71BTgMfe9CTK3uwN4B7hbVafkNdfh2m1ouWcSxw+nUraCVRhTSORiSyLdmYULibjhBkhOhmLFYNYsr987uU6dOnTs2JGpU6fSvXt39u3bx/z583n//fcz2kyYMIEpU6awe/duEhMTSUlJyfhivJATJ06wf/9+YjOtR1BQEG3btuWPP/7IGLdjxw6efvppvv/+e/78809cLhcul4vff/89x+ty8uRJ9u3bR7t27bKMb9++PV9++WWWcU2aNMl4XKlSJQAOHTqUo+Xs3LmTlJSULMsJDg4mNjY248ZrAwYMoFu3btStW5fu3btz7bXXcs011xAUFETTpk3p2rUrjRo1onv37nTt2pXevXtzySWX5HhdC4Kvj8GMB5KB8kA/4C0Raeih3SCgJ9AUaAL0AO7J3EBEooEngE35DZVwWRtKkMip1fmelTEBzdW2LSxaBKNGOT+9XFzSDRw4kE8++YSjR48yffp0YmJiuPHGGwGYM2cOQ4YMYcCAAcyfP5/169dz//33k5ycXKAZrr/+ev78808mTpzI999/z48//khISEiBLSd7F/6hoaF/mXa+40q5XU6LFi3YtWsXL7zwAi6Xi/79+9OtWzdcLhfBwcF88803fPPNNzRp0oS3336bOnXq8NNPP+V7+bnhswIjIhFAL+BpVU1Q1eXAPOB2D837A+NUdY+q7gXGAQOytXkBeB1nSyhfUps7PSu7vl+T31kZE/hiY+Hxx31WXAB69+5NWFgYM2fOZOrUqdxxxx0ZX8DLly+nbdu2DB48mBYtWnDppZeyY8eOHM+7VKlSVKxYMcsxBlVl9eqzJ+4cOXKEX375hSeeeIKuXbty2WWXER8fT2pqakab9GMu2U8OyCwqKopKlSqxYsWKLOOXL19OgwYNcpz5QmrWrEmxYsWyLCctLY1Vq1ZlWU7JkiXp3bs3b731Fl988QWLFy9m+/btgFOIYmNjefbZZ1mzZg2VKlVizpw5BZYxJ3y5P6gukKqq2zKN+wno6KFtQ/e0zO0ytnREpA3QCrgf+Pv5Fioig3C2iChfvjxxcXF/abMpqSTNKM3JhV8T1+3S865EQkKCx3n4QyBlgcDKczFmKVWqFPHx8fmaR1paWr7nkVe9e/fm2Wef5fjx4/Tp04f4+HjS0tKoVq0a06dP56OPPqJWrVp89NFHfPvtt5QuXToja1JSEi6X65zP7733Xl566SWqVq1Kw4YNmTx5Mvv376dcuXLEx8cTEhJCmTJlePPNN4mOjmb//v089dRThISEcObMGeLj44mJiSE8PJx58+ZRtmxZihcvTqlSpf6yrAceeIDnn3+eKlWq0KxZM+bMmcOyZctYtmwZ8fHxJCQ4NzE8derUX97rxMTEc77/KSkppKamEh8fT1hYGAMHDmT48OGUKFGCGjVqMH78eA4ePMgdd9xBfHw8b7zxBuXLl6dJkyaEhIQwffp0oqKiKFWqFIsWLSIuLo4uXbpQrlw5fv75Z/744w9q1qzpcflnzpzxzt9LXu+1nNsB6AAcyDbubiDOQ9s0oH6m53UABQQIBtYCl7unxQF35SRDy5Ytz3E/atWv6a5Hqzc9532p012M93ovKIGU52LMcr77pueUr+/1ntm6desU0CuuuCJLnqSkJL3zzju1dOnSWqpUKb3zzjv1ueee0+rVq2e0e/bZZ7Vhw4bnfJ6SkqJDhgzRUqVKaalSpXTw4MF67733aseOHTPaLFq0SBs2bKjFixfXhg0b6tdff60RERE6bdq0jCyTJ0/WqlWralBQUMZrsy8rLS1NR44cqVWqVNHQ0FBt1KiRzp07N2P6zp07FdA1a9ZkWX9AP/zww3O+P/3799frrrsuI8uZM2f0oYce0nLlymmxYsW0bdu2umzZsoz2kyZN0ubNm2tkZKSWLFlSr7zySl2xYoWqOp+Vq6++OuO1tWvX1jFjxpxz2ef7bAFrNa/f+3l9Ya4XBM2B09nGPQx85qHtCaBNpuctgXj34weAqZmm5bvA7N+vOoonNS0oWPXUqXO+0aoX5xdXQQmkPBdjlsJeYDwJpDxFOYu3CowvD/JvA0JEpE6mcU3xfJB+k3uap3ZdgJtE5ICIHACuAMaJyBt5DRYdDatpQ5ArDX78Ma+zMcYYk4nPCoyqngI+BkaKSISItANuBN710HwGMFREKotIJZwtnenuaQOAy4Bm7mEt8BzwZF6zFS8Om8KdA/2stiv6jTGmIPj6oo/7ganAIeAIcJ+qbhKRDsBXqpp+ietEoBawwf18inscqno88wxFJBk4qaon8hMsuUxFjhyrShkrMMYYUyB8WmBU9SjO9S3Zxy8DIjM9V2C4e7jQPDsVRLboaNiW1ppYKzDGGFMgrLNLt+ho+CmsDfz2Gxw54u84xnikareVMAXLm58pKzBuMTGwWp2elVljF1yawBMaGkpiYqK/Y5iLTEpKCiFe6oTRCoxbdDSsTGoJInag3wSkcuXKsXfvXk6fPm1bMqZAuFwuDh48SCkvdXhqPTu6RUfDHyeioH5924IxASkqKgo421NuXpw5c4awsLCCjJUvgZSnqGaJiIjI6Em6oFmBcUtIgNOn4UC1NlRY/RWoOlszxgSQqKiojEKTF3FxcTRv3rwAE+VPIOWxLAXPdpHh3Ngv/YZ7LyxqA4cOQS668DbGGPNXVmCAuDhI70D1uzS74NIYYwqCFRigUyfnan6ADdIEV2gxOw5jjDH5ZAUG57YYixZBmzbgCi2ONm1mWzDGGJNPVmDcYmPhpZcgKQm2x7SBtWvP7jczxhiTa1ZgMunQASpXhi8OtYZTp2DLFn9HMsaYQssKTCZBQdCnD0zbaFf0G2NMflmByebWW2FTal2Sw6LsOIwxxuSDFZhsWrSAS+sEsSG8tRUYY4zJBysw2Yg4WzELjrVGf/4ZzpzxdyRjjCmUrMB4cOut8D1tkNRUWL/e33GMMaZQsgLjQf36cLqh+0C/7SYzxpg8sQJzDl37V2YfFYlfZAXGGGPywgrMOfzjH7CaNiSvsFOVjTEmL6zAnEO1anCoehvKHNmGHj3m7zjGGFPoWIE5jwo3OsdhfvtgrZ+TGGNM4WMF5jyueKAVANvfs+MwxhiTW1ZgzqPspaXZU6IusnYNLpe/0xhjTOFiBeYCkpu1oXHi96xaqf6OYowxhYoVmAuo1LMNFTnAV1P2+juKMcYUKlZgLiCsg3ML5b1zV5Oa6ucwxhhTiFiBuZBmzXAFh1D35BoWL/Z3GGOMKTyswFxIWBg0bcoVwauZPdvfYYwxpvCwApMDQW3b0CZoDXM/cpGcbG+ZMcbkhH1b5kTr1oSnxFMxfivffRfj7zTGGFMoWIHJiTbOFf1dSq5h8eJyfg5jjDGFgxWYnKhfHyIjuaXGalatKsPJk/4OZIwxgc8KTE4EB0PLlrRIXU1ycjCffOLvQMYYE/h8WmBEJEZE5orIKRHZLSJ9z9FORGSMiBxxD2NERNzTyorICvf44yKySkTaeT18mzZE7viJquVO2NlkxhiTA77eghkPJAPlgX7AWyLS0EO7QUBPoCnQBOgB3OOelgDcCVwCRANjgM9EJMSrydu0QZKTuaPZUhYsgD//9OrSjDGm0PNZgRGRCKAX8LSqJqjqcmAecLuH5v2Bcaq6R1X3AuOAAQCqekZVt6qqCxAgDafQePf0LveB/mvKLiMtDf7v/7y6NGOMKfRE1TedOIpIc2CFqpbING4Y0FFVe2RrewLorqrfu5+3ApaoaslMbX4G6gOhwBRVvfscyx2Es0VE+fLlW77//vt5WwFV2t14I6cuuYQ7T09mR7kWvPba+rzNq4AkJCQQGRnp1wyZBVIey+JZIGWBwMpjWTzr3LnzOlVtlacXq6pPBqADcCDbuLuBOA9t04D6mZ7XARR3Qcw0Pgy4FeifkwwtW7bUPFu5UjUoSF2gySHhejkrdffuvM+uICxZssS/AbIJpDyWxbNAyqIaWHksi2fAWs3j974vj8EkAFHZxkUB8TloGwUkuFc2gzq7y2YDj4lI04IM+xdxcaCKACGuJDoRx5w5Xl2iMcYUar4sMNuAEBGpk2lcU2CTh7ab3NMu1C5dKFAr3wnPp1MnCAtzNqOAQ/U72tlkxhhzHj4rMKp6CvgYGCkiEe5Ti28E3vXQfAYwVEQqi0gl4GFgOoCIXC4i7UWkmIiEi8ijOGelfe/VFYiNhUWLONyuHbhc9Gh/jB9/hF9+8epSjTGm0PL1acr3A+HAIWA2cJ+qbhKRDiKSkKndROAzYAOwEfjCPQ6gOM7pzkeAvcC1wHWqus/r6WNj2TxiBNSqxXWrniIIl23FGGPMOfi0wKjqUVXtqaoRqlpNVd9zj1+mqpGZ2qmqDlfVGPcwPP34i6p+q6pNVbWke1pHVV3qs3UICYHnniN003qeafgRs2eDj07EM8aYQsW6ismLW2+FBg146PDT/PZrKj/84O9AxhgTeKzA5EVwMIwaRemDW+kfPNN2kxljjAdWYPLqppugZUueL/YcH81OxuXydyBjjAks+S4wIhJaEEEKHREYPZryibu4Zt8Uli3zdyBjjAksuSowIvKgiPTK9PxtIFFEtopIvQJPF+iuuoq0KzrwtIzm/2ac9ncaY4wJKLndgnkQ+BNARK4E/g70BdbjdEhZtIgQ/OJ/qKj7iXn/TVJS/B3IGGMCR24LTGVgp/txD+BDVf0AGAFcXoC5Co8OHTjU/CoePP0CSz61W10aY0y63BaYk0D6Tem7AYvcj1NwOp4skqLHj6YMR4kf+Yq/oxhjTMDIbYH5BpgsIlOAS4Gv3OMbcnbLpsgJjW3FD9VvotuGcZz+44i/4xhjTEDIbYH5F7AC526SvVX1qHt8C5yuX4qslGdGEUkCvw9+yd9RjDEmIOTqNsOqehJ4wMP4ZwssUSHVqn9DPh7cjx5f/A/2D4GKFf0dyRhj/Cq3pyk3yHw6soh0E5GZIvK4iAQXfLzCIzgYfvnHCILSUkh69nl/xzHGGL/L7S6yqUBzABGpCnwKxODsOhtdsNEKn+731WYqdxI0ZSI/fLzL33GMMcavcltg6gPpXTv2Br5X1WuB23FuXVykpabCaJ4mTYPY+PeRrFrl70TGGOM/uS0wwUCy+3EX4Ev34x04N/0q0r79FvYFVeEt7qNf2jv8/OFWf0cyxhi/yW2B2QjcJyIdcArM1+7xlYHDBRmsMOrUCYoXhxd5nETC6RRX5M99MMYUYbktMI8CdwNxwGxV3eAefwOwugBzFUruuypz/4hyTC81hHo/zuHwop/8HcsYY/wit6cpLxWRS4AoVT2WadJEwHp7xCkysbGwudswjrUbz699nibm4DyC7MYIxpgiJtdfe6qahtODciMRaSgiYaq6S1UPeSFfodXgitJs7/kIsYc/Y+bg7/wdxxhjfC6318GEiMjLwDHgJ2ADcExEXiqy94U5j1YzHuR48XJUeetJO6PMGFPk5HYL5iXgNuBeoC5QB7gP5zTlFwo2WuEnJSMp/uzj/I3FvN5zMceOXfg1xhhzschtgekLDFTVd1R1h3uYDtwF9CvwdBeB8H/fS1K5Kjx06PHJmjcAAB2tSURBVEnuGqio+juRMcb4Rm4LTCmca16y2wGUzn+ci1BYGMVHP8PlfEfS3C+YMMHfgYwxxjdyW2B+wrmrZXYPuacZTwYMQGvX5rWSTzF0iIuf7J0yxhQBuS0ww4H+IrJVRN5xD1txjssMK/h4F4nQUOS556gd/xP9S3zIP/4Bp075O5QxxnhXrgqMqi7FObj/f0Cke/gQuArPWzYmXZ8+0LAh4yKfYcfWVB74y00PjDHm4pKX62D2qeqTqtrLPTwFnAJ6FXy8i0hwMIweTcSebcy5/l2mTYNZs/wdyhhjvMeuL/elG2+E1q25ae0TTKo2iql3r+LXX/0dyhhjvMMKjC+JQN++yIED3PXHs3yW2IX/XL+KpCR/BzPGmIJnBcbXTjtdtokq4ZJExW1xPPqonzMZY4wX5KizSxGZd4EmUQWQpWjo3BnCwyExEVEXnVuc4KrX4G9/gxtu8Hc4Y4wpODntTflIDqbvzGeWoiG9T/+FC+Grr+i+agwjqtXln/+8k59+gipV/B3QGGMKRo4KjKr+09tBipT0Pv0feQR69uSZb+5ib2gwffv2Z/FiCMnVTRSMMSYw2TEYfwoLg7lzka5dmZjyT6otm8moUf4OZYwxBcOnBUZEYkRkroicEpHdItL3HO1ERMaIyBH3MEZExD2troh8KiJ/ishREZkvIvV8uR4FKjwcPvkE6dyZGdKf7SPfY8kSf4cyxpj88/UWzHggGSiP0/vyWyLS0EO7QUBPoCnQBOgB3OOeVhqYB9Rzz2c18Kl3Y3tZiRLw2We42l/JDG5nzs1z+PNPf4cyxpj88VmBEZEInKv9n1bVBFVdjlMobvfQvD8wTlX3qOpeYBwwAEBVV6vq26p6VFVTgFeAeiJSxicr4i0lShDy1ecktmjPG8f78Wq7D3n+eexGZcaYQkvURzcoEZHmwApVLZFp3DCgo6r2yNb2BNBdVb93P28FLFHVkh7m2xN4S1UrnmO5g3C2iChfvnzL999/P1/rkZCQQGRkZL7mcT7BiYlccvszXHrkR/7OHL4s3pNx436iYcOTPs+SW4GUx7J4FkhZILDyWBbPOnfuvE5VW+XpxarqkwHoABzINu5uIM5D2zSgfqbndQDFXRAzja8C7AVuzUmGli1ban4tWbIk3/O4kJefOakriNVkQvQGPtHhw/2XJTcCKY9l8SyQsqgGVh7L4hmwVvP4ve/LYzAJ/PWCzCggPgdto4AE98oCICKXAN8Ab6rq7ALO6lftri7JTWFfs46WfMgt7H7jMxYv9ncqY4zJHV8WmG1AiIjUyTSuKbDJQ9tN7mke24lINE5xmaeq//FCVr+KjYVPFkex8pn5xNduxozE3vy365eMHYvdctkYU2j4rMCo6ingY2CkiESISDvgRuBdD81nAENFpLKIVAIeBqYDiEgUMB/neM5jPgnvB7GxMPS5UpRZ+w3BzRozV25i4SNf06cPJCT4O50xxlyYr09Tvh8IBw4Bs4H7VHWTiHQQkcxfmxOBz4ANwEbgC/c4gJuA1sA/RSQh01DNZ2vhS6VLE7zwG0KaNuTzkJ6c+PAbLr8c6+bfGBPwfFpg1Dm1uKeqRqhqNVV9zz1+mapGZmqnqjpcVWPcw/D04y+q+o6qinsekZmG3325Lj4VE4MsWEBIw/p8EXojdX9fSOvWsGpV4T4z2xhzcbOuYgqLMmVg4UKC69Xh/1Ju4B/llvDEE40ZMQJcLn+HM8aYv7ICU5iULQuLFhFUuxYTfr+GLyrfxvznVnHDDXD8uL/DGWNMVlZgCptLLoExY5DkZK7ZO4vlwR058dUKWreGjRv9Hc4YY86yAlMY/fwzBAUhQHBaCgvK9aPi8S20bQtz5vg7nDHGOKzAFEadOkGxYriCgqBYMcJOHeXbk814vcxz3NEniWHDIDXV3yGNMUWdFZjCyH1XzF133glxcbB9O9KrFwP/GMGumBasHLeS7t2xHpmNMX5lBaawio3l9379nGJTrhy89x588QUVIxNYIe35+9J/cXmDk/zrX9YjszHGP6zAXEyuvRY2bUIefJB7XG/x7eEG7HnzUzp0gA8+8Hc4Y0xRYwXmYhMZCa++yjv3fsdRYviUnryf1psh/9hP//7WA4AxxneswFyk6t3ehvZh63hK/sP1fM6voZdRcvYkLqvnskJjjPEJKzAXqdhYmL84lIj/PMGW938mol1z3ki5h62VO7N+zlbq18cKjTHGq6zAXMRiY+Hxx6H5P+rC4sUwZQq1E35mvTbhi8tH8ckHyVZojDFeYwWmqBCBgQNhyxakZ0+uXvkMR6q3YHaXyVR/7wUG1FtlhcYYU6CswBQ1FSo4l/vPm0fI0UP8fcEgnkt9krigv7H7/VW2RWOMKTBWYIqqHj3g/vtBBEEJTTvDwvJ9mX7tB8z9ICVLoVm1Cl54wa6nMcbkjhWYouyqqyAsDIKDITSUkNQkbv/8HxwrXYMv2z7Hsg/2U68edOgATz0FXbpYkTHG5JwVmKLM3eUMo0bBt9/CH3/AZ58R3KwJV60awY7Uaiwo04fL05bjcimJifDYY7B2LTi3fzPGmHOzAlPUpZ9qFhvrbMlcfz189RVs24Y88AAdk+aznA6spxn3yCR+XH6K1q2hZk0YNgy++85ueGaM8cwKjPGsTh34738J2b+HHY9OokJFYYLew/HIymzs/m+61/yV11936lL16jBkCGzYEGXFxhiTwQqMOb+ICGq/eDfl9/4Iy5YRdO01NFz8BpPi6nLqyqtZPPRzWjVP48c3V7HvwfncWG4VDzzg7HFLS/N3eGOMP4X4O4ApJESgfXtn+O9/YfJkQidMoPOiHnSuUAHVI6ikkXK8OFdNWsQbb8RSrhzcfDPccguEhsLy5c6tbGJj/b0yxhhfsC0Yk3sVK8Izz8Du3U43zeHhSGoKQeqieFoii5r8m+XD5nLtFceZMcM5++zKK+HJJ50CM3u2HbcxpiiwAmPyLjTU2TyZNQvCwlARCAoieONPtBt7M9M+K8vJxlfwUeNnacdygjWF5GTo2xdiYqBbN+f053nz4OBBf6+MMaag2S4yk3+xsbB4MTunTqXWnXdCy5bO6WULFhC8YAE3bRrNzYzkJCVZGtQJ7dqdtdHd+GxrXV58UTKO1VSrBm3bQps2zs8WLSAiwq9rZozJByswpmDExvJ7UhK10g+wXHmlM4wahRw7xta3FnPq0wV03bOAsG8+owfwXNWqpPbrxo5a3VgsXdi3dDslF8Tx0YedeIRYgoOhUaOsRef4cVi2zI7lGFMYWIEx3hcdTb0nesETvZznv/0GCxbAggWEzPuYesenUg+cEwmAR0JDWX//ROaF/YOVP4bzwQcwaVLWWQYHO13ZdOwItWo5Q4UKEGQ7fY0JGFZgjO/VqgX33OMMaWlO1wBPPQULFwIgKck0f+2fNA8aCJdeinZuxNFKjXhnXSOmfNeIX7mU1LRQpk6FqVPPzjYszLkAtFYtKFbsUtavdx7Xru2ML1HibNtVqyAuzraEjPEmKzDGv4KDnX1fI0fCihWQnOycPPDUU87jTZuQjRsp8+knDHW5GAokUYytUp/yXRoR3KQRv0c1YnNQI346Xp0dO4OI3LCKmrsXM2fu3/iOs9WjQgWn4JQs6fSQk5bmLOrFF+GKKyA62jn5oHRpCMnBX4YVKWPOzwqMCQzp/aKd6xv7zBn45Rd+nbuRw3EbqZ+6kehfV8DC9ygLtABui4hwzhTYuQ11uRgROpLd/xrD5krd+OVkJTbvK81vO4Xvv4fUVGe2yckwdOhf45QsebbgREf/9fGxY/Daa858QkNh/HjnOFFYmDOEh599fCEFVagKYj6WxbvzCaQsBTmfc7ECYwJHbOy5P+VhYdCsGXWaNaNO5vEnT8LmzbBxozN8/jmkpSEAKcnUePXf1ACuBedbv1IlTlavxPz4SuxxVeZgcCW69a9ERJ1KHAipzH4q8eepEhw9CjHbVlFjZxwrDndi+cFYjh2Do0chKSlrtKQkuOuuc69WUFBHSpQ4W3AyF6DkZNiwwbkuKCjIuY61bFlnwy442BmX+ee5xh044FySlJbmPO/XD6pWddqInB12767OihXO48zTgoKcy5omTnSKZkiIczeHGjVy/2vctQvefPPsfO67z+lOCM52kpr+c/v2qqxZ89fxu3bB22+fncfAgc76ZJa9w9XMz9Mf//47TJ9+9n0ZMODsfLK/fteu6nz7reeOXP/4A2bMODuf/v2d+aS/f3D+n857n3Wd7rrL+X/IU/4dO6qxcqXn9fr9d5g2Les6ZZ5PTqW/Ny4XFC/u/H9X0EVGtAh1i9uqVStdu3ZtvuYRFxdHp06dCiZQPgVSFgiQPKtWQZcuuJKSCCpWDF55xdnk2Lfv7LB3L4m/7SP4wF6KpZz+6zxKlXJe8/vvzl9fcDBcd52zf61ECVKKRfDLngjeml6Ck2kRJAWX4O//jKBs9QhOU4JTRHCKCOLTShC1ewPh6z7j4GU38Ut0LGfOkGX45RfYufPsoitUgDJlnC8Pl+vsz6anV9EmMY6VoZ1YHRz7l+lnzpzdKoOzX26Z/7wvZxWdiCOOTll2HeZWIM2noLIUlEBap9zMJzjY6VT98cezTVi1iipXXLF3j2qVvGSwLRhzcXHvatuVfk3OOf4lCwfn2/fkySyFJ+Px4sVnuxtIS3OeL1kCp04R6nLRGHgzfWapwORzR1JAfnrZKVqlSztnG4SHQ4kSHC8Xzre7S5CgJUgOCufqDiWoWMuZltFu3z4YO/bsv75PPOF0RupyOevgcrF9m4uxLyuuVBehIcrQf7uoXdOZpi5Fd+xAxr+RMY+0e/+F1qiFIs4gws5dwltvCSmpQnCIcN/9Qs1akmUTKGjnDoq//jKkpkBIKGeGPIqr1qVZ1ve3HWd3H4aEOB2h1qp1tugBBP22nWKvjMmYT/Kwx+DSugBIkLB9O4x5yckSEgKPPS7UqSuI4FzQC8iv2wj+z8iz6zRiFNKggbM55t4827I1iOGPBZGcGkRQSBDjXgmiYeOgjDYS7H68cQN7P/2Uytf3gMaNEXU576972LTBxcP/dmW8vy+96OKyei40LVO7LZsJeu7Zs3kefQKtUcv5/KSl8dt2F+NfT0NT0ygWksZ9g1xUr5Lm/i/B+Sm/7yb43eln5zHgTqhREwkJydhc3fl7MK+NDyEpNRgJCWbo8BDq1HemaXCw86Zv+5WgZ59yzycU139egMsuy/J72vILPPkEznscDDeGAfMzNdi8GR5/nApQ+dyf7vOzLZhcCoj/0t0CKQsEVp58Z3FvCZGcDMWKnd1/oOqMO3UKTp/O+jP743nznFsfqDrfrq1aQb16TpvExIyfpw+fJul4IhFBp50tqsREZxnGBIBWwFpVuWBDD3y6BSMiMcDbQHfgMPC4qr7noZ0ALwLpe7anAI+puxqKyCSgI1AHuFNVp3s/vSlSznXSgYizw7p4ceeo//k0aQJLlji764oXd/6t97BFVcI9ZJGWdrYILV/uHFRJSXHOKJg61ektIf0gSk5+rl0LvXqdzfLxx07BU83dsG4d3Hbb2Szvvut0uZBbP/wAt9+OJicjxYrBO+9A8+ZZD8ac63H6z/Xr4e67nSwhITBhgnNlbqYtj/QtvPMOc+Y43R2lHwi77Tbo0+fsllDmIf399DRs2OCcep9+5se0ac57nH6gLPuBM0/PV6+Gbt3O/p6+/hpat3Y+D6mpGVtDf3mefdoPPzgH0dJ/T//7HzRu/Nffw/k2MDZsgAcfRJOS8r4Voqo+G4DZwBwgEmgPnAAaemh3D7AVqIKzebYZuDfT9H8BXYC1wICcLr9ly5aaX0uWLMn3PApKIGVRDaw8AZNl5UrdcdddqitX5ns++vzz+ZtPIGUpqDwF9L5oeLimBQWphocHRJ6A+T2tXKmVYY/m9Ts/ry/M9YIgAkgG6mYa9y7wooe2K4FBmZ4PBL7z0G65FZjAEUh5LItngZRFNYDyFNSXegEJmPdFVYG1msfvfZ8dgxGR5sAKVS2RadwwoKOq9sjW9gTQXVW/dz9vBSxR1ZLZ2i0Hpuh5dpGJyCBgEED58uVbvv/++/laj4SEBCIjI/M1j4ISSFkgsPJYFs8CKQsEVh7L4lnnzp3XqWqrPL04r5UptwPQATiQbdzdQJyHtmlA/UzP6+A+GSdbO9uCCSCBlMeyeBZIWVQDK49l8Yx8bMH4smvABCAq27goID4HbaOABPfKGmOMKQR8WWC2ASEikvlC7KbAJg9tN7mnXaidMcaYAOWzAqOqp4CPgZEiEiEi7YAbcQ70ZzcDGCoilUWkEvAwMD19oogUE5EwQIBQEQkTEeuo3RhjAoivv5Tvx7mI+hDOKcv3qeomEekgIgmZ2k0EPgM2ABuBL9zj0n0DJAJXAJPcj6/0fnxjjDE55dMLLVX1KNDTw/hlONfGpD9XYLh78DSfTl6KaIwxpoDYbiVjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV/i0wIhIjIjMFZFTIrJbRPqeo52IyBgROeIexoiIZJreTETWichp989mvlsLY4wxOeHrLZjxQDJQHugHvCUiDT20GwT0BJoCTYAewD0AIlIM+BSYCUQD7wCfuscbY4wJED4rMCISAfQCnlbVBFVdDswDbvfQvD8wTlX3qOpeYBwwwD2tExACvKqqSar6OiDA37y8CsYYY3IhxIfLqgukquq2TON+Ajp6aNvQPS1zu4aZpv2sqppp+s/u8V9nn5GIDMLZIgJIEJGteYufoSxwOJ/zKCiBlAUCK49l8SyQskBg5bEsntXL6wt9WWAigZPZxp0ASp6j7Yls7SLdx2GyTzvffFDVScCkvAT2RETWqmqrgppffgRSFgisPJbFs0DKAoGVx7J4JiJr8/paXx6DSQCiso2LAuJz0DYKSHBvteRmPsYYY/zElwVmGxAiInUyjWsKbPLQdpN7mqd2m4Ammc8qwzkRwNN8jDHG+InPCoyqngI+BkaKSISItANuBN710HwGMFREKotIJeBhYLp7WhyQBjwoIsVFZLB7/GJv5s+kwHa3FYBAygKBlceyeBZIWSCw8lgWz/KcRbIeK/cuEYkBpgLdgCPAY6r6noh0AL5S1Uh3OwHGAHe5XzoFeDT9wL6INHePawBsAQaq6o8+WxFjjDEX5NMCY4wxpuiwrmKMMcZ4hRUYY4wxXmEFJgfcJxO87e4/LV5E1ovINQGQq46InBGRmQGQpY+IbHH3M7fDfVzNHzlqiMiXInJMRA6IyBsi4pPrvURksIisFZEkEZmebVoXEfnF3X/eEhGp7o8sInK5iCwQkaMi8qeIfCgiFf2RJVubZ0RERaSrN7NcKI+IlBCRN0XksIicEJGlfszyd/ffVLyIbBaRnl7Oct7vubx8hq3A5EwI8AdOrwOlgKeAD0Skhh8zgdO32xo/Z0BEuuGclPFPnAterwR+81OcN4FDQEWgGc7v7H4fLXsfMBrnRJYMIlIW5wzKp4EYYC0wxx9ZcPrvmwTUAKrjXD82zU9ZABCR2sAtwH4v58hJnkk4v6PL3D//7Y8sIlIZp7/FoTjX+T0CvCci5byY5Zzfc3n9DPvySv5Cy32K9YhMoz4XkZ1AS2CXPzKJSB/gOLASuNQfGTJ5Dhipqt+5n+/1Y5aawBuqegY4ICJfc7abIa9S1Y8BRKQVUCXTpJuBTar6oXv6COCwiNRX1V98mUVVv8rcTkTeAL71RoYLZclkPPAozj8HXneuPCJSH7gBqKKq6b2OrPNHFvfj45l+X1+IyCmgNs4/UN7Icr7vuTLk4TNsWzB5ICLlcfpW88vFnSISBYzE+e/Gr0QkGGgFXCIi20Vkj3u3VLifIr0K9HHv6qgMXIOHPup8LEvfeu4/5B34qPBdwJX48SJlEbkFSFLVL/2VIZM2wG7gOfcusg0i0stPWdYCW0TkBhEJdu8eS8Lpd9Ensn3P5ekzbAUml0QkFJgFvOOt/z5zYBTwtqru8dPyMysPhAK9gQ44u6Wa42xe+8NSnA/9SWAPzh/qJ37Kki5X/ef5iog0AZ7B2f3ij+WXBJ4HHvLH8j2oAjTC+d1UAgYD74jIZb4OoqppOBecv4dTWN4D7nF/sXudh++5PH2GrcDkgogE4fQ8kIzz4fNHhmZAV+AVfyzfg0T3z/+p6n5VPQz8F7jW10Hcv5+vcfYVR+D0SBuNc3zInwKu/zwRuRT4CnhIVZf5KcYI4F1V3eWn5WeXCKQAo1U1WVW/BZYA3X0dxH2yw0s4tycphnNcZIr44OaK5/iey9Nn2ApMDomIAG/j/MfeS1VT/BSlE84B2t9F5AAwDOglIj/4I4yqHsPZUsh8xa6/rt6NAarhHINJUtUjOAewfV7sssnSt54490aqjf92sVYHFgKjVNVTV02+0gWny6cD7s9yVZyDyo/6KY+n3U/++iw3A5aq6lpVdanqGuB7nH8uveY833N5+gxbgcm5t3DOLOmhqokXauxFk3B+sc3cwwTgC+AqP2aaBjwgIuVEJBrnzJvPfR3CvfW0E7hPREJEpDTOzet8st/avcwwIBgIFpEwcU6Rngs0EpFe7unP4NzTyGu7WM+VxX1cajFOEZ7greXnJAtOgWnE2c/yPpw71473U56lwO/A4+427YDOwHw/ZFkDdEjfYhGne6wOeP+zfK7vubx9hlXVhgsMOKdzKnAGZ1MxfegXANlGADP9nCEU5wyg48AB4HUgzE9ZmuF0iHoM54ZNHwDlffi70GzDCPe0rsAvOLth4oAa/sgCPOt+nPlznOCv9yVbu11AVz//nhoCq4BTwGbgJj9mGQxsx9kN9RvwsJeznPd7Li+fYeuLzBhjjFfYLjJjjDFeYQXGGGOMV1iBMcYY4xVWYIwxxniFFRhjjDFeYQXGGGOMV1iBMaaQEOd+Kb39ncOYnLICY0wOiMh09xd89uG7C7/amKLJ7gdjTM4tBG7PNi7ZH0GMKQxsC8aYnEtS1QPZhqOQsftqsIh84b6l7G4RuS3zi0WksYgsFJFEcW5ZPF1ESmVr0999H5IkETkoIu9kyxAjzm2OT4nIbx6W8Yx72UnuTiRneOWdMCYHrMAYU3CeA+bh9Ic2CZjhvlNheu+z83H6dmoD3ARcQaZb5YrIPcBEnM5Dm+D0Ar0x2zKeAT7F6dl2DjBVRKq5X98Lp3ft+4E6wPXAai+spzE5Yn2RGZMDIjIduA2nI8DMxqvqoyKiwBRVvTvTaxYCB1T1NhG5GxiLczveePf0Tjj3G6mjqttFZA9Ox6WPnSODAi+q6uPu5yE4N1YbpKozRWQoTm/EjdR/t5MwJoMdgzEm55YCg7KNO57p8aps01YB17kfX4bTvXnmGzStBFxAAxE5CVQGFl0gQ0Z37aqaKiJ/AuXcoz7EuTvkThGZj3PztXmqmnSBeRrjFbaLzJicO62q27MNhwtgvrnZjZB9y0Rx/x2r6h9APZytmJPAOGCde/ecMT5nBcaYgnO5h+db3I+3AI3d96FPdwXO3+AWVT0E7MW5CVeeqeoZVf1CVf8NtMa5v0m7/MzTmLyyXWTG5FxxEamQbVyaqv7pfnyziKzBuRlTb5xi0dY9bRbOSQAzROQZIBrngP7Hqrrd3eY/wCsichDnLqUlgC6qOi4n4URkAM7f9Pc4JxP8A2eL59dcrqcxBcIKjDE51xXYn23cXqCK+/EIoBfOHT3/BP6pzr3UUdXTInIV8CrOmV1ncM4Geyh9Rqr6logkAw8DY4CjwJe5yHcceBTnZIJQnDsy3qyqO3MxD2MKjJ1FZkwBcJ/hdYuq/p+/sxgTKOwYjDHGGK+wAmOMMcYrbBeZMcYYr7AtGGOMMV5hBcYYY4xXWIExxhjjFVZgjDHGeIUVGGOMMV7x/zEmCMRVB6lFAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "-r6pZD_05jk8", "outputId": "4be1cf45-3e15-4119-e0dc-b52eb6601d20", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVyVVf7A8c/3sgqoIAIuKKgI7hsuuCWa2mo1LVbaNk3ZtEwz07TNNJXT1NRU069ZypbJaSqrsdLWyS0hl9xARUVwQ1CRRRZBNoF7z++Pe2EuBMhyVzjv1+u+9J7nPOee+3Dhe5+zilIKTdM0TbM3g7MroGmapnUNOuBomqZpDqEDjqZpmuYQOuBomqZpDqEDjqZpmuYQOuBomqZpDqEDjqZpmuYQLhlwROQBEUkSkfMi8u4F8v5aRHJFpFRElouIj9WxSBFJEJEKEUkXkbl2r7ymaZrWJJcMOMBp4FlgeUuZROQS4HHgYiACGAz8wSrLR8AeIBh4AvhURELsUWFN0zStZeLKKw2IyLNAuFLqjmaOfwhkKqV+Z3l+MbBCKdVHRKKB/UBvpdQ5y/HNluNvOOQNaJqmafU8nV2BDhoJfGH1PAUIE5Fgy7GMumBjdXxkUwWJyBJgCYCvr2/swIED7VNjN2IymTAYXPUm2HH0ddDXoI6+Di1fg8OHDxcopZptRXL3gBMAlFg9r/t/9yaO1R3v31RBSqm3gLcAYmJi1KFDh2xbUzeUmJhIfHy8s6vhdPo66GtQR1+Hlq+BiGS1dK67h+oyoIfV87r/n2viWN3xc2iapmkO5+4BJxUYa/V8LJCnlCq0HBssIt0bHU91YP00TdM0C5cMOCLiKSK+gAfgISK+ItJU8997wM9EZISIBAK/B94FUEodBvYCT1vO/wkwBvjMIW9C0zRNa8AlAw7mwFGJecjzLZb//15EBopImYgMBFBKrQFeBBKAE0AW8LRVOTcBE4Fi4AXgeqXUGYe9C03TNK2eSw4aUEotBZY2czigUd5XgFeaKScTiLddzTRN07T2csmA4+pKS0vJz8+npqbG2VWxq549e5KWlubsajiMv78/4eHhXX7Yq6bZiw44bVRaWkpeXh79+/enW7duiIizq2Q3586do3v37hfO2AmYTCays7MpKCggNDTU2dXRtE5Jf5Vro/z8fPr374+fn1+nDjZdjcFgICwsjJKSxlO3NE2zFR1w2qimpoZu3bo5uxqaHXh5eVFbW+vsamhap6UDTjvoO5vOSf9cNc2+dMDRNE3THEIHHE3TNM0hdMDRNE3THEIHnC5ARFp83HHHHU6rW2RkJC+//LLTXl/TNMfRAcdJVqyAyEgwGMz/rlhhv9fKycmpf7z99ts/SvvrX//apvKqq6vtUU1N0zo5HXCcYMUKWLIEsrJAKfO/S5bYL+j06dOn/hEYGNggrby8nNtuu40+ffrg7+/PhAkT+PrrrxucHxkZydKlS7nzzjsJDAxk8eLFACxfvpyBAwfi5+fHggULeP3113800uurr74iNjYWX19fBg0axBNPPFEfsOLj48nKyuKRRx6pv9vSNK3z0gHHCZ54AioqGqZVVJjTHa2srIzLLruM9evXk5KSwnXXXce1115Lenp6g3yvvPIKw4YNIykpiT/96U9s27aNu+66i/vvv5+9e/dy1VVX8fTTTzc4Z+3atSxevJgHHniA1NRUli9fzqeffsrvfvc7AFatWkV4eDhPPfVU/d2WpmmdmFJKPxo9oqOjVXMOHjzY7LHWElHKfG/T8CHS4aIv6JNPPlHmH3vzpkyZov74xz+q0tJSpZRSERER6sorr2yQ56abblKXXHJJg7S77767QdkzZ85UzzzzTIM8q1evVv7+/spkMtWX/dJLL7X7/dhaUz/fhIQEx1fExehrYKavQ8vXAEhSLfxt1Xc4TjBwYNvS7am8vJxHH32UESNGEBQUREBAAElJSZw4caJBvokTJzZ4np6ezuTJkxukTZkypcHz5ORknnvuOQICAuofixYtory8nNzcXPu8IU3TXJZevNMJnnvO3Gdj3azm52dOd7SHH36YNWvW8PLLLzN06FD8/Py47bbbfjQwwN/fv81lm0wmnn76aW644YYfHQsJCWl3nTVNc0864DiBpc+dJ56AEyfMdzbPPfe/dEfasmULt912G9dddx0AVVVVHDt2jOjo6BbPGzZsGLt27WqQtnPnzgbPJ0yYQHp6OlFRUc2W4+3tjdFobGftNU1zJzrgOMnixc4JMI1FR0ezevVqrr76ary8vPjDH/5AVVXVBc978MEHmTFjBi+99BLXXHMNmzZtYvXq1Q3yPPXUU1x55ZVERESwcOFCPD09OXDgADt37uTFF18EzCPgNm/ezC233IKPjw+9e/e2y/vUNM35XLYPR0R6ichqESkXkSwRWdRMvm8t207XPapFZL/V8UwRqbQ6vs5x78L1vfLKK4SGhjJz5kwuu+wy4uLimDlz5gXPmzp1Km+//TZ/+9vfGDNmDJ9//jmPPfYYvr6+9XkuueQSvvnmGxISEpg8eTKTJ0/mhRdeYKBVZ9UzzzzDyZMnGTJkiG5m07ROzpXvcF4DqoEwYBzwjYikKKVSrTMppS6zfi4iicDGRmUtUEptsGNd3cb111+PeTCJWUREBBs2NLw0Dz/8MGDegA0gMzOzybLuvPNO7rzzzvrnv/71r3/UfDZ//nzmz5/fbH3i4uJISUlp03vQNM09uWTAERF/4DpglFKqDNgiIl8CtwKPt3BeJDATuMP+tdReeukl5s2bR0BAABs2bOCNN97gT3/6k7OrpWmai3LJgANEA7VKqcNWaSnArAucdxuwWSmV2Sh9hYgYgD3AI0op/ZXaBpKSknj55ZcpKSlh0KBBPP/88/zyl790drU0TXNRYt284ipEZCbwiVKqj1Xa3cBipVR8C+cdBZ5VSr1rlTYd2A0I8EvLY5hS6myjc5cASwBCQkJiV65c2eRr9OzZs8VRV52J0WjEw8PD2dVwqKNHj/5om+mysjICAgIapB0priWt0MSIYA+igjr/NWrqGnRF+jq0fA1mz56drJSa2ORBXPcOpwzo0SitB3CuuRNEZAbQB/jUOl0ptdXq6fMicjvmZrevGuV7C3gLICYmRsXHxzf5OmlpaXTv3r1Vb8LdnTt3rsu81zq+vr6MHz++QVpiYiLWn4fkrGKeX/sDJgX/zTKy4q44YiOCHFxTx2p8DboqfR06dg1cdZTaYcBTRIZapY0FUpvJD3A7sMrS59MShfluR9PaZfXuU5gsDQPna0xszyh0boU0zU24ZMBRSpUDq4BnRMTf0ix2NfB+U/lFpBuwEHi3UfpAEZkuIt4i4isijwC9ga1NFKNprZJy6n+tsQoY3rdr3QVqWnu5ZMCxuA/oBuQDHwH3KqVSRWSmiDS+i7kGOAskNErvDiwDioFs4FLgMqWU/kqqtcvWowXszy7lp9MjuXPGIDwE1qXmObtamuYWXLUPB6VUEeZA0jh9MxDQKO0jzEGpcd5UYIy96qh1LUopXlp7iL49fXns0mH4enlgAN7Zepxb4iIY1b+ns6uoaS7Nle9wNM2lbEjLZ+/Js/zy4qH4eplHpj04dyi9/LxZ+mUqrjjiU9NciQ44jvTii5DQuNWvkYQEcz439emnnzbYufPdd9/t8DDSxMRERISCgoKOVq/dTCbFy2sPMai3P9fFhten9/D14pFLYkjKKubLlNNOq5+muQMdcBxp0iRYuLD5oJOQYD4+aZLNX/qOO+6o38bZy8uLwYMH8/DDD1NeXm7z17J24403kpGR0er8kZGRvPzyyw3Spk2bRk5ODsHBwbauXqt9te80h/LO8et50Xh5NPy1uWHiAEb178EL36ZTUV3rpBpqmuvTAceRZs+GlSubDjp1wWblSnM+O5g7dy45OTlkZGTw7LPP8vrrr9evm2attrbWZs1D3bp1IzQ0tENleHt706dPnwZ3To5Ua1K8sv4ww/p058rRfX903MMgLF0wkpySKt5IPOaEGmqae9ABx9GaCjoOCDYAPj4+9OnThwEDBrBo0SIWL17M559/ztKlSxk1ahTvvvsuQ4YMwcfHh/LyckpKSliyZAmhoaF0796dWbNmkZSU1KDM9957j4iICPz8/LjyyivJy2s4YqupJrX//ve/TJkyhW7duhEcHMyCBQuoqqoiPj6erKwsHnnkkfq7MWi6SW3VqlWMHj0aHx8fBgwYwHPPPdcgSEZGRvLss89yzz330KNHD8LDw3nppZfadd22ZNeSVVjBI5fEYDA0HfQmRvbiqrH9eHNTBieLKprMo2ldnQ44zmAddJ56yiHBpindunWjpqYGgOPHj/Phhx/yySefkJKSgo+PDzfccAPZ2dl8/fXX7Nmzh4suuog5c+aQk5MDwI4dO7jjjjtYsmQJe/fuZcGCBTz11FMtvuaaNWu46qqrmDdvHsnJySQkJDBr1ixMJhOrVq0iPDycp556ipycnPrXaSw5OZkbbriBa6+9lv379/PCCy/w/PPP849//KNBvv/7v/9j9OjR7N69m8cee4xHH32Ubdu2tekaVdUY+eJoDRMGBjJnWMt3ar+9fBgGER79NIXXEo6SnFXcptfStE5PKaUfjR7R0dGqOQcPHmz2WJs9+aRSYP7Xzm6//XZ1xRVX1D/fsWOHCg4OVgsXLlRPP/208vT0VLm5ufXHv/vuO+Xv768qKioalDN27Fj15z//WSml1M0336zmzp3b4PjPfvYzZf5Ymf3rX/9S/v7+9c+nTZumbrzxxmbrGRERoV566aUGaQkJCQpQZ86cUUoptWjRIjV79uwGeZ5++mnVv3//BuXcdNNNDfJERUWpP/7xj82+tlI//vk+/cV+FfHY1+rdrcdbPK/O45+mqIjHvlaRj32tYn7/X5WUWdSq81xdQkKCs6vgEvR1aPkaAEmqhb+t+g7HWRISYNkyePJJ878XGr1mA2vWrCEgIABfX1+mTp3KRRddxN///ncAwsPDCQsLq8+bnJxMRUUFISEhBAQE1D8OHDjAsWPmfoq0tDSmTp3a4DUaP29sz549XHzxxR16H2lpaUyfPr1B2owZM8jOzqa0tLQ+bcyYhlOw+vXrR35+fqtfZ8vRAt79IQuA579Na9UdS5+e5g3oFHXL3jhvZJ2muRqXnfjZqTXus5k92yHNahdddBFvvfUWXl5e9OvXDy8vr/pj/v7+DfKaTCZCQ0PZsmXLj8rp0aPxuqquw3pggfX7qztmMplaXdZHO07U/7+m1rxm2oUW6ZwxNIRliceoqjWhMC/0WV1rwttTf7fTNP1b4GhNDRBoafSaDfn5+REVFUVERMSP/hg3NmHCBPLz8zEYDERFRTV41I06Gz58ONu3b29wXuPnjY0fP57vvvuu2ePe3t4YjcYWyxg+fDhbtzZcDm/Lli2Eh4fbdHXrkkpz/5YB8PI0EDf4wsOyYyOCWHF3HI9cEs0NE8PZmH6G25bv4GxFtc3qpWm2kJxZxN83HnFoX6O+w3GklkajWQcdJwwgaGzu3LnExcVx9dVX8+KLLzJs2DByc3NZs2YNc+fOZebMmTz44INMmzaN559/nuuvv57ExERWr17dYrlPPPEECxYsICoqikWLFqGUYt26ddxzzz34+fkRGRnJ5s2bueWWW/Dx8aF3794/KuM3v/kNkyZNYunSpSxatIhdu3bxl7/8xaa7jdYaTezPLiE+OoTe6iw3z53U6i0IYiOC6vNOH9KbRz/dx09e/4F3bp/I4JCuvZeK5njJWcVszygkbnAw/QJ92Xq0kK9Ssvn+sLm591U5wv/dNJarxva3e130HY4j7drVcjCpCzq7djm2Xk0QET799FPmzJnD3XffTUxMDAsXLuTQoUP069cPgLi4ON555x2WLVvGmDFjWLVqFUuXLm2x3Msvv5zVq1fz7bffMn78eGbNmkVCQgIGg/mj+Mwzz3Dy5EmGDBlCSEhIk2VMmDCBTz75hM8++4xRo0bx+OOP8/jjj/PAAw/Y7P3vOXmWksoabpg4gCuHeLd7v5trxvfnw7unUFJZw4K/b+Gxz/bp0WuawyRnFXPzW9t5ae0hrl/2A1Of38jDn6Sw8/j/PoNGpXjwo73c834Se07Y97Ppkjt+OltMTIw6dOhQk8fS0tIYPny4g2vkHF1xA7a6n++f16Tz9qYMdj81j93bt3Z4063/7svh/g93owBfL4PbbdqmNx4zc7fr8PLaQ/wj4Wj989kxITx66TDKz9dyyzs7qKk14elhYMHYvqw/mE9JZQ1TBvVi7ogwqmuNxA3u/aPPaUvXQETccsdPTXOqhPR8JkYG0cO35b6u1jpe+L8lhKpbOQBB0zqqbgyNQcDb08ADc4YyvK950M+Ku+Lqm9piI4IoO1/LxztP8HrCMXYcLwLA1+uoTb8c6SY1TWsk+2wl6bnnLjjRsy3iBgfXj1QziLRqAIKmddTR/DKC/Lx4aF70jwJHbEQQ98+Oqk8L8PHkrpmDuWN6ZP2WyD/dspKTn33T8ou0YcFhHXA0rZGEdPNcHVsGnNiIID68awp+3h5MjwrWdzea3VXVGPn+8BkuG92XB+YMbfVnbnpUb3y8DHgIHAyP4YrnfmWzBYd1k5qmNZKQns+AXt0YYuMRZbGRvZg2JJiMAvuu0K1pANsyCqmoNjJvRNiFM1uJjQiyam6bhtedU5oePduONSB1wGkHpZTTVi7W7Me8/AZsPVbAjRMH2OVnPH5gEBvS8impqKGnn236hzStKRsO5uHn7cHUdjTfWg/tJ6LRlA2Rdi847LJNaiLSS0RWi0i5iGSJyKJm8i0VkRoRKbN6DLY6Pk5EkkWkwvLvuI7Uy8vLi8rKyo4UobmompoaqoyKqhoT8TZsTrM2fkAgAHtPnbVL+ZoG5i9PG9LyuGhoSP3utB1iNU8wcvnyds8XdNmAA7wGVANhwGJgmYiMbCbvf5RSAVaPDAAR8Qa+AD4AgoB/A19Y0tslNDSU7OxsKioq9JbCnYjJZCIvL4/9+TX4ehna9a2wNUaH90QEu8930Lq2/dkl5JWeb3NzWotmz4Z77yXy/ffh3nvbNTndJZvURMQfuA4YpZQqA7aIyJfArcDjbSgqHvN7fNWykunfRORhYA6wpj11q1tH7PTp0/VL+3dWVVVV+Pr6OrsaDuPv78+/dhcyfUhv23wrbEJ3Xy+iQ7uz54S+w9HsZ8PBPAwCs215p25ZcDjz1luJXLbsf+tAtoFLBhwgGqhVSh22SksBZjWTf4GIFAE5wD+UUsss6SOBfarhrcg+S3qDgCMiS4AlACEhISQmJnb4Tbi7srKyH22e1pmdLjORWVjJRWHGBj//srIym34ewrzOk3T8HAkJCW7TF2jra+Cu3OU6rN5VSVSggX27frBJeYF79jDiD3/g4NNPc2roUM6OH8+In/yEg08/zdnx41tfUEt7FzjrAcwEchul3Q0kNpF3BNAP8ACmYQ46N1uOPQl83Cj/CmBpS6/f0n44XUlX2/vjze+PqojHvlanihvuAWTr6/DRjiwV8djX6lj+OZuWa09d7bPQHHe4DieLylXEY1+rN78/apsCN25Uqndv87/K6ho0SlfKfffDKQMar4HfAzjXOKNS6qBS6rRSyqiU+gH4K3B9W8vRtI3p+Qzr053+gd3s+jrjBloGDpzUzWqa7W04aN7mfd6IPh0vrLULDrdylXtXDTiHAU8RGWqVNhZIbcW5CuonyqYCY6Rhu8WYVpajdSGlVTUkZRbbts27GUNDu+Pv7aH7cTS72JCWz5AQfwb19r9w5gux8YLDLhlwlFLlwCrgGRHxF5HpwNXA+43zisjVIhIkZpOBBzGPTANIBIzAgyLiIyJ1ywlvtPub0NzK5sMF1JqUTVcXaI6HQRg7IJA9J/VINc22Sqtq2J5RyFxbjU579NELDwyYPducrxVcMuBY3Ad0A/KBj4B7lVKpIjJTRMqs8t0EHMXcTPYe8Gel1L8BlFLVwDXAbcBZ4E7gGku6ptXbmJ5Pz25e9fNk7G3cgEDSc85RWd3yZnOa1hbfHzpDrUkxb7gNh0PbkKuOUkMpVYQ5WDRO3wwEWD2/+QLl7AFibV5BrdMwmRTfH85nVnQInh6O+Q42fmAQtSbFgdMlTIrs5ZDX1Dq/9QfzCPb3ZvxA11yrz5XvcDTNIfZnl1BQVs3sYU1v+GYP4+pWHND9OJqN1BhNJBzKZ86wUDwMrjncXgccrctbsSMLgEC/di9A0WYh3X0ID+qm+3E0m9l1vIhzVbW2679pZMUKiIyEOXNmERlpft5WOuBoXVpyVjGfJJ0C4N4Pkh26/fP4gUH6DkezmfVpefh4Gpg5tLfNy16xApYsgawsUErIyjI/b2vQ0QFH69K+3JtN3TIUNZadOB1l3IBATpdUkVtS5bDX1DonpRTrD+YxI6o3ft6275p/4gmoqGiYVlFhTm8LHXC0Lu1M2XkAPAS8PA0O3YlzfP0EUN2spnXM6j3ZnCquZGiYfZaiOnGibenN0QFH67KqaoxsPVrI9KjePDQ/xqZ7t7fGiL498PIQ9ugVB7QOSM4q5tFP9wHwr62ZdmkWHjiwbenN0QFH67LWH8yjpLKGn88a3GBvd0fx9fJgRL+eesUBrUO2ZxRSazI3DNca7dMs/Nxz4OfXMM3Pz5zeFjrgaF3WyqST9A/sxrQhtu9kba3xAwLZf6qEWqPJaXXQ3NvkQeZ5XIL9moUXL4a33oKICBBRRESYny9e3LZydMDRuqTss5VsOVrAdbHhTp2zMH5gIJU1Rg7l6fVktfbx8zbv3XT56L52bRZevBgyM2Hjxu/JzGx7sAEdcLQu6rPkUygFN8SGO7Ue4weY/zjoZjWtver6bB6/bJjDm4XbSgccrcsxmRQrk04yPSqYAb38LnyCHQ3o1Y1e/t56qwKt3ZIyiwnrYZ5I7Op0wNG6nO0ZhZwqrmThxAHOrgoiwvgBgew5oYdGa+2TnFXMxIhebrF7rA44WpezMukk3X09uWSkDTaosoFxAwI5dqacksoaZ1dFczM5JZVkn610+aa0OjrgaF1KSWUN3x7I5epx/fD18nB2dQDqV/Z99uuDDl1aR3N/SZnmz8vESB1wNM3lfJlymvO1Jm6c2MYZa3akLIvrfJp8isX/3K6DjtZqyVnFdPPyYHjfHs6uSqvogKN1KZ8knWRYn+6M6u86v6D7TpUA5r3RHb2em+bekrKKGDcgEC8H7ePUUe5RS02zgbScUvadKmHhxAEu1cEaNzgYT8tcIE8Px67nprmvsvO1HDxdyiQ3aU4DFw44ItJLRFaLSLmIZInIombyPSIiB0TknIgcF5FHGh3PFJFKESmzPNY55h1ormZl0km8PIRrxvd3dlUaiI0I4pWFYwG4dWqE23QAa86198RZTApi3WjHWJcNOMBrQDUQBiwGlonIyCbyCXAbEARcCjwgIjc1yrNAKRVgecy3Z6U120nOKua1hKM26dPYnlHARztPMCmiF738HbfRWmtdNa4/g0P8Sc/RKw5orZOUVYTI/1YddwcuGXBExB+4DnhSKVWmlNoCfAnc2jivUupFpdRupVStUuoQ8AUw3bE11mxtR0YhN765jb+sO9ThjvRv9+ew6O0dVNWYSMoqctlO+fkj+rA9o5CSCj08Wruw5KxiYsK608PXy9lVaTVRSl04l4OJyHhgq1LKzyrtYWCWUmpBC+cJsBt4Uyn1hiUtE+iGObjuAR5RSqU0ce4SYAlASEhI7MqVK233htxUWVkZAQG22V/jaLGR9CIjw3p5EBXU8nDkWpPiya2V5JSbP5sG4NqhXlw5pG13JmXViq+OVbM+q5a6pTHbU5Ytr0NLjp418uz2KpaM8WFaP9tvotURjroGrs5VroNJKe7bUMG0fp7cNtLHoa/d0jWYPXt2slJqYnPnutan+n8CgNJGaSVA9wuctxTz35R/WaUtxhyEBPglsFZEhimlGqwlopR6C3gLICYmRsXHx7e37p1GYmIitrgOO48X8sK6HZiUwtvT2OICgzVGE/ev2E1OeQUeBsFoUogIN8+d1Kq+jeSsYrYcOUP+ufN8mXKa8vO1xA8LYevRQmqNJrw8Da0uq46trsOFXGRSvJn6HadUEPHxsXZ/vbZw1DVwda5yHVJPl1C1dgtXTRtFvIP7JDtyDVw14JQBjcet9gCabeAWkQcw9+XMVEqdr0tXSm21yva8iNwOzAS+sl11tZa8s+V4/X4d1bUmtmcUNPkHv8Zo4hcf7mHdwTyWLhjB6PBAXlqbzvaMIk63YjZ1clYxN721jRqj+bUmRgTxp2tHEx3WneSsYrZnFBI3ONhlO+UNBmHeiDA+35NNVY3RZSamaq6nrlnYVT/LzXHJPhzgMOApIkOt0sYCqU1lFpE7gceBi5VSpy5QtsJ8t6M5gNGk2HeqpP6CmxRsPVrI2YrqBvlqjCYe/GgPa1JzeerKEdwxfRCxEUG8/7MpTBgYyO9W7edEYcWPX6CuXJPiz2vS6oONQWD2sFCiw8w3xbERQU7ZZK2t5o8Io6LayA/HCpxdFc2FudOCndZcMuAopcqBVcAzIuIvItOBq4H3G+cVkcXAn4B5SqmMRscGish0EfEWEV/LkOnewNbG5Wj2sf5gLjklVfx6XjQPz49m0ZSB7DxexCWvbuL7w2cA8y6Fv/p4L98eyOX3VwznzhmD6s/38jDw15vGg8AvPt5DTRMblVXXmnho5V52Hi/GwyB4CHjbaSMqe5s6JJgAH0/WpeY5uyqaC3OnBTutuWqTGsB9wHIgHygE7lVKpYrITOBbpVRdr9WzQDCwy+rif6CU+jnmPp9lwBCgCtgLXKaU0lO5HUApxbLvMxjYy4/7Z0fVb3S2aPJAfv2fvdy+fCeXjerD8YJy0nPP8fsrhnPXzME/KmdALz/+fN0Y7luxm5fXHeK3lw2vP3auqoaff5DM1qOFPHJJDHGDerH9eJFLN521xMfTg/iYEDak5WE0KaduDqe5ptNnzQt23jVz0IUzuxiXDThKqSLgmibSN2MeVFD3vNmrrpRKBcbYpYLaBe04XkTKybP88ZpRDf5wjurfk69+MYOHP0nh6305AHgapH4Ry6ZcProvN08eyB/4M80AACAASURBVJvfZzB9SG8uig4hr7SK25fv5Gh+GX+5YSzXWTZTc6eJcE2ZNyKMr/flsPdkMbER7v1eNNtLsvTfTHTDz4ZLNqlpncMb3x+jd4B3k7tq+loWHKyLQ0qpC64h9tSVI4gOC+AXH+3msc9SuOJvmzlZVMHyOybVB5vOYPawULw8RDeraU1KzizCz9uD4X0vNGjX9eiAo9lFWk4piYfOcMe0yGZHW8UNDsbb04CHgFcr+ly6eXtwX3wUJZW1/GfXKQrKqll61Uguig6xx1twmh6+XsQNDmZtai6uOE9Oc66krGLGDQjE000W7LTmfjXW3MJbmzLw8/bglriIZvPERgSx4q44Hpof0+LcHGvZZyvrR7wZBPLPnW8xv7uaP7IPmYUVHM0vc3ZVNBdSdr6WtJxSJrph/yTogKPZwaniCr5MOc3NkwcS6NfyjP62DleOGxyMj5fBrUeitca84WEArDuom9W0/3HHBTutueygAc19vbPlOAL8bIbtR9HU3RW5+iTOjurT05exAwJZl5rL/bOjnF0dzUW444Kd1mx+hyMifxORr5tI7yEiS0VkuFXar0Rkv4joO61Oori8mo93nuSqcf3oF2ifSWnuMomzo+aPCCPlVAm5JVXOrormIhLS8+nt782RPPdsarXpH3oRGQL8HPOaZo1NBJ4GrJc2fRMIAW63ZT0053l/exaVNUbuuWiIs6vi9i4ZaW5WW5+mm9U02JVZRMqpEs6UVbvtVuS2vrP4FZCilEpq4th44DxwsC5BKVUJvAc8bON6aE6w7VgByxKPERsRREwf9xuy6WqGhATQr6cv72zOcMs/Lpptfb4nu/7/7roVeasCjohEiUiNiDzTKH2ZZafNiSLiA9wCfNjE+WnAy4APUCMiSkQ+sxz+GBghItM69E40p0rOKubWd3ZSWWNk/6kS/QfSBnafOEveufNkFlaw6G33/Ear2U5FtRGg1dMIXFGrAo5S6ijwT+BXIhIMICJPAXcCP7Hc0cQBgcDmJoq4DcjAvELzVMvjIcuxvZhXgb60/W9Dc7btGYX1K0IbTe757cvVbM8orJ+Hc77WxLrUXCfXSHOm/dkljO7fo03TCFxNW5rUngE8gMdF5C7M/TG3KqU2WI7HYV6JeV8T56YA4cBGpdR2yyMLQCllshyPa+d70FxA3axnwX2/fbmauomxdasxfJVymsKyzjnvSGvZ8YJyjuaXce2EcLceMNPqYdFKqRwReRX4jeW8B5VS1tti9gNKlVLVTZw+EvDGvBFaU84A0a2ti+Z6zlXVAnBLXATXjO/vtr8QrsR6CHjPbl788euD/PTdXXx4dxwBPnpGQ1eywTIfa65lfpa7auun9gjmfpgtSqnXGh3zxTwooCkTMN/97G3meCXmbaA1N7X5SAGBfl4svWqkXuHYhmIjguqDd79AX+5+L5kl7yXxr59OwsdTb9DWVaxPy2NYn+4M6OXn7Kp0SKub1ETkYszDmLcB00Wk8SrMhZj7cJoyHjimlGq8bXSdXoDeccpNKaXYcqSA6UN662BjR3OGhfHS9WP44Vghv/p4L0aTXmetKygqryYps4j5I9z77gZaP0ptArAa88CBeOAE8HyjbOmAt4g0tWzvCKyGQzdhEHCoNXXRXM/R/DJyS6uYObS3s6vS6V07IZzfXzGcbw/k8vMPknkt4YgevdbJbUzPx6Rg3og+zq5Kh10w4IhIFPAtsA74haWP5g/A5SJykVXWTZZ/JzdRzFlgrIhcIiJxdSPdLOUHYu6/2dTEeZob2HzEfHM6Qwcch7hr5mB+Mr4/6w/m8dLaw9z45jY+2nmC6lrzbqjJWcW8lnBUB6JOYsPBPPr08GVU/x7OrkqHtdiHIyJ9MAeaNGCxZUQZmCdrPgq8AEwDUEplishOYAHm7aGtPQW8A3yOua9nJrDFcuwKoBrzHZTmhjYfOcOg3v6EB7l3+7I7iQr1RzB3jNaaFL9dtZ8/fJXKkBB/DuWWYTQpvDwM/P7K4ZZ9hwSDwJG8cxw4Xcr4AUGMDu+Jl4fgYRC8PAwczCklNbuEqUN6Nxj0kZxVzNfHquk+qFgPBnGwqhojm46c4doJ/d1uO+mmtBhwlFK5wI/2/FVKGYHhPz6DZcBfReR+pVSFVf4DwJRmXuYW4JPG2z6LSC/MQWo+5v6d3yqlmppUKpgD312WpH8CjyvLBAYRGWcpZzjmwPkzpVRzgxe0Njpfa2R7RhE3TOw8G6C5g7jBvfHxOkpNrQlPDwO/vDiKgrIa/rv/dP18qGqjiae+SG3y/Pe2ZbVQ+mGiQvwZHBKAQaR+u+uvM7e77fwPd/XDsQIqqo2dojkNbL9a9AfAY8B9mFcWaJElGMzBPGy6sdcw3/mEAeOAb0QkxbJttLUlmLeiHov5C9964Djwhoh4A18ArwKvA/cAX4jI0GaGb2tttDvrLJU1RmZE6eY0R2pu1ewrxvRl8dvbqTaa8DQY+N0Vw4gK6Y5RKb7cm82q3dkozHsJXT66L7NjQjGaFBvS8lh/MI+6YQi1JkVWYQVZReX1AaxuORUdcBxn/cF8Anw8iRvsntsRNGbTgKOUqhWRn2IeBt0afYA7LCsZ1BMRf+A6YJRSqgzYIiJfArcCjzcq43bgL0qpU5Zz/wLcDbyBeYCDJ/Cq5Y7nbyLyMOYgt6Ydb1FrZMvRM3gYhKlD9ERPR7MeMm2dtuLuprdvCPDx5Jv9OdTUmvDyNPDT6YPqjw8JDWDTkTP1x/6ycByxEUEkZxVz89vbqa41YRDRE3odyGT5IjArOqTzDIFXSrncA/Mw6opGaQ8DXzWRtwSYYvV8InDO8v9fA982yv818JsmylkCJAFJfn5+CvPdkn5c4NHn1ldU2OI/O70e+tG6h3e/YapH3A3Ku9+wVh/z7jdM9b//fdX3ztecXv+u9PDuG60iHvta+Y+Id3pd2vBIaulvu6vuQxMANJ6zUwI0tQRxgOWYdb4AS99O42PNlqOUekspNVEpNTE8PNzpQdcVHgkJCS0eLyo7j2//aB6/4xqn19WZ18GdHuez0yjZtpLz2WmtPnY+O41bYsPwDongcG6p099DV/ksPPuvL/AwCKd2rXX6+27tNbgQVw04ZUDjMYA9MC/yeaG8PYAyZX73bSlHa6MfjhWiFMwcGuLsqmh2Nq2fJ54G4dPkU86uSpex/mAekyN7XXCbdnfiqgHnMOApIkOt0sYCTQ25SbUcaypfKjBGGo4nHNNMOVobbT5yhu6+nowN7+nsqmh21sNHmD0slFV7sqk1mi58gtYhWYXlHM4rY24nWF3AmksGHKVUOea5PM+IiL+ITAeuBt5vIvt7wEMi0l9E+mFeXPRdy7FEwAg8KCI+IvKAJX2jPevfFSil2HykgGlDgvH0cMmPkWZjN8SGc+bceb4/fMbZVen01lsW6+wMy9lYc+W/FPdhXtAzH/gIuFcplSoiM0XEekPvNzHvs7MfOAB8Y0lDmYc+X4N5P56zmPfvuUbpIdEddrygnOyzlczQzWldxuxhoQT7e/NJkm5Ws7cNnWSxzsZcdo1zpVQR5mDROH0z5sEAdc8V5lUPHm2mnD1ArJ2q2WVtOWpezmamnn/TZXh5GLhmfH/e25ZJUXk1vfw7T9+CKzlbUc2uzGLunTXE2VWxOVe+w9Fc2KbDBQzo1Y2I4M71DUxr2Q0Tw6kxKj7fk+3sqnRay7ccx2hSDOjV+XZs0QFHa7Mao3nG+YyokE6xvpPWesP69GB0/556tJqdJGcV848E8zz4p79M7XQLsOqAo7VZysmzlJ2v5SK9OnSXdMPEcPNCn6cbT3HTOmrLkTPUbXNUt5RQZ6IDjtZmm44UYBCYNkQHnK7oqrH98PYw6MEDduDrZV7CxiDg5WnodEsJ6YCjtdmWI2cYHR5ITz8vZ1dFc4JAP2/mjQjji73Z9XvwaLaRcaYcPy8PfjU3ulOuzK0DjtYmmw6fYc+JswwN9Xd2VTQnun5iOMUVNXyXlufsqnQadat2zx0RxoMXD+10wQZ0wNHaIDmrmJ/9excK+HJvTqfr0NRa76KhIYT18OGTTjZ4oKXdUus2orPX5373iWIKy6uZP7JzTfa05rLzcDTXsz2jkFqjuUfTaNJ7o3RlHgbh2gnhvJF4jBfXpHPx8DC3/ywkZxWz+O3tnK814ekh/PLioQzqbZ7yl3GmjL9tPEKt0X4b0a09kIu3h4H4mFCblutKdMDRWi1ucDB1+xp3xg5NrW1G9O2OApYlHmP51uNu3+ewPaOAKkufVI1R8fK6w03ms8dGdEop1h3MY3pUMAE+nffPcud9Z5rNjejbAwMwcVAvHr10mFv/cdE67kRRJWDeBKUz7AZaWlkLmL9TeXsaeOHaMYzo1wMROHi6lEc+TaHGqBA7bER3KO8cJ4oquDe+860uYE0HHK3V9p48i1HBPbMGu/UfFs024gYH42EQjCaFl4d73/FmFZbz/vYsRvfvwSUj+zB1SO8Gn/HoMPO6Zg99sJ0T50yYw6ztrEvNQwQuHt55m9NADxrQ2iA5qwiACQN1sNHM21k/eeVwAH45131HVRlNioc/ScFDhDdvncgDc5p+L7ERQTw+xZfwXt345cd7OVdVY7M6rE3NZcLAIEK7+9qsTFekA47Warsyi4kOC+hUG0JpHbNocgT+3h6cKq50dlXa7Z0tGezKLGbpVSPpF9jy+mXdPIVXbxxPTkkVT39hm221ThVXkHq6lEs68ei0OjrgaK1iNCl2ZxUzMbKXs6uiuRBvTwPTonqTeOhMq7YYdjWHcs/x8trDzB8RxrUT+rfqnNiIIH4xJ4pVe7L5Ym/HFzGt2/tm3og+HS7L1emAo7XKodxznDtfy6RI92w20ewnPiaE7LOVHDtTduHMLqS61sRDK/fS3deTP107uk0L0T4wO4oJAwP5/ecHOFVc0aF6rEvNIzosgEG9O/9kah1wtFZJsvTfTIzQdzhaQ3XzRhIPuddOoP/YeITU06U895PR9A7wadO5nh4GXr1xPErBQ/9JwWhq391dcXk1OzOLmN8F7m5ABxytlZIyi+nTw5fwoM63R4fWMf0DuzE0NMCttp7+z64T/D3hKLOie3PpqPb9sR8Y7MczV49kZ2YRd/xrZ7tWIPguPR+jSXXq1QWsuVzAEZFeIrJaRMpFJEtEFrWQ9xEROSAi50TkuIg80uh4pohUikiZ5bHO/u+gc0rKLCI2Mkjvf6M1aVZ0CDsyiqiornV2VS5o5/FCHv9sP0rB9oyiDi1VE9HLD4PA5iMF3PjmNna0cTuBdam59O3py+j+PdtdB3ficgEHeA2oBsKAxcAyERnZTF4BbgOCgEuBB0TkpkZ5FiilAiyP+faqdGeWfbaS0yVVTHLTYa+a/cXHhFJtNLHtmOvv3/LZ7lP1s2hqjR3bc2b78aL6/9eaFPd/uJt9p8626tzKaiObjpxh/oiwLvNFzqUCjoj4A9cBTyqlypRSW4AvgVubyq+UelEptVspVauUOgR8AUx3XI27hqRMS/+NHqGmNWPSoCD8vD3coh+nrMp8F+Zhgz1n4gYH4+1pMJflYZ4E+5PXf+DFNelU1RhbPHfTkTNU1ZiYP7Jr9N8AiCsNZRSR8cBWpZSfVdrDwCyl1IILnCvAbuBNpdQblrRMoBvmwLoHeEQpldLM+UuAJQAhISGxK1eu7PgbasHRYiPpRUaG9fIgKsjDrq/VXmVlZQQEBPBe6nl+OF3Laxf74WHoGt/ErNVdh66sNdfg1eQqsstMvHhRN5f9xq6U4pFNlQT6CONCPNr8+9fUdbD+Xe4bYODj9Go2Z9fSz1+4JNKLc9Wqydd5e9959uTX8rc5fni60e9VS5+F2bNnJyulJjZ3rqstbRMAlDZKKwG6t+LcpZgDy7+s0hZjDkIC/BJYKyLDlFI/uudVSr0FvAUQExOj4uPj21r3Jm06dIZvDpymb89uBPh4kn/uPOk5pWw5WoBJgbenkY/uds1FDxMTE4mPj+eFvZuYNNiHi+dMcXaVnKLuOnRlrbkGJ32zePLzA0SMmsTgENcM0Gk5pRSs3cxDl45i0ZSBbT6/qesQ3yjPFfMg8VA+v1mZwr9SqwHwkFoWjO3H6PCehHT3oXeANykJSUSG9CBoyCiX/P1vTkd+HxwacEQkEZjVzOGtwC+AHo3SewDnLlDuA5j7cmYqpc7XpSultlple15EbgdmAl+1rebt8999Odz34e4GaT6eBny9DPX7llfXmvjPrhMu+4ErqazhUN45Lh/d19lV0VxcfHQIYB4e7aoBZ4NlkuVcO69ZFh8Tyi1xA/nbd0dRgFEpvkzJ5vNGE0XTckpZ/E/7bHfgihzah6OUildKSTOPGcBhwFNEhlqdNhZodg0JEbkTeBy4WCl1od2gFOa7HbsrKq/mic/31z83CDw4J4r0P17K8jsm4+tlbvcVYNXubL7dn+OIarXZ7hPFKAUT9YRP7QIG9PJjcIg/iS48PHp9Wh7jBgQS2sP+a5ZdFB2Kj+X33NfLwCf3TGXvU/NY/+uLuHFieN1OH/UrbXcFLjVoQClVDqwCnhERfxGZDlwNvN9UfhFZDPwJmKeUymh0bKCITBcRbxHxtQyZ7o35TsquKquN3PnuLsrO1+LtYf7AeXsamBUTiogQGxHEirvieGh+DP/+6WTGDgjk/g9389HOE/auWpslZRbhYRDGDQh0dlU0NxAfHcr2jEIqq1vuMHeG3JIq9p0qYd4Ix8x5sf49X3FXHLGRvQj082ZoWHcWThpYH4y60t5SrtaHA3AfsBzIBwqBe5VSqQAiMhP4VilVd7/+LBAM7LLqpPxAKfVzzP0+y4AhQBWwF7hMKWXXrxK1RhO/+GgPKafOsmxxLCHdfdieUUjc4OAGt8yxEUH1zycOCuK+Fbv57ar9FFdUc++sIS7T6bors5hR/Xrg5+2KHxXN1cTHhLB863G2Hy9ktovtXLk+zdycNt9BAQca/p43Tl9xV1yTfxs6M5f7K6KUKgKuaebYZswDC+qeD2qhnFRgjM0r2AKlFE9+kcqGtDyeuXpk/QzmC32Y/Lw9efu2iTz8SQovrjlEek4pMX26Eze4t1M/iLUmRcrJs9wSF+G0OmjuZfKgXvh6Gfj+0BnXCzgH84gI9iMq1DX6l5oLRp2ZywUcd/aPjUf5aOcJ7osfwm1TI9t0rpeHgf9bOI7qWhNfpuQgKTn4eB11amdiZqmJ87UmvWCn1mq+Xh5MHRxM4qF8oLn52o53rqqGbccKuH1qpMu0HnRFLtWH466Ss4q594Nk/rL+MNeO788jl8S0qxyDQRjV3zxIzxU6E48Um/d3j9ULdmptEB8TSmZhBZkF5c6uSr1NhwuoMSqH9d9oTdN3OB2UnFXMzW9tp9powiBw46QBHfoGFTe4Nz6eRzlfa7LL3ultcbjYyKDe/oR0b9tKulrXFh9jHh79/eEzRLrIkvvrD+YS5OfV5ZqwXI2+w+mghPR8qo3mOwEBkjqwECCY23U/vDuOiF5+BHbzctroMKUUR4qN+hdUa7OIYH/69vTl3z9kdmhhTFupMZrYmJ7PnGFheHroP3nOpK9+Byil2JVpbvIy2HB4Y2xEEL+9fBgF5dVsTM/vcHntcexMOWU16P4brc2Ss4rJP3eejIJyFr+93elBZ9fxIkqrapk3wrUGMXRFOuA0oeS8atUvycqkk+w4XsxtUyP4Td1YexvdEcwdHkZYDx8+2J5lk/LaKjlLL9iptc/2jML67abPu8CkxvVpeXh7Gpg5NMSp9dB0wGlS8XnFogt8M8s4U8bSLw8ybUgwSxeM5P7ZUTZtfvL0MHDTpIFsOnKGE4Ud28K2PdYcyMXbAGfLqx3+2pp7q1tBuY4z75KVUqw/mMeMqN74++gua2fTAacZ52tNfL4nu8ljNUYTv/rPXrw9DbyycBwGO630evPkgRhEWLHTsXc5yZlFJB46Q7UJFr+zw+lNIpp7qZvUeM24fiigoMx5X1rSc89xqrhSj05zETrgNEOAj3adYOWukz869uqGw+w7VcIL146mT0/7rcnUp6cvc4eH8knSKc7XOm6pkK/25dRvUOXsodmae4qNCOIvC8cREezH25szLnyCnay3LNZ5sZ0X69RaRwecJgT5CMt/Oom4QcE8+tk+nli9n+pa80i0HRmFvJ54jBsnDuAyB6ygfEtcBEXl1Xy7P9fur1WnbptgA11rnSfNtjwMws9mDGLPibP1fYKOtqFusc7u9l+sU7swHXCa0NNHmB0Tyrs/ncQ9swazYscJbn57O1/uzebu95II6+7DUwtGOKQu04f0JjLYz6GDB/adKmF43+5cO9SryyybrtnH9bHh9OzmxT83H3f4a69PzWPfqRJG9Gu844nmLDrgtMDTw8BvLxvOa4smcCC7hAc/3ktpVS1FFTWk57a4RY/NGAzC4ikRJGUVk57beG862zteUE567jluiB3AlUO8dbDROsTP25PFUwayNjXXoYNfkrOKuXdFMgCfJZ/S/ZAuQgecVrhiTF8WTf7f7oBGo2P7Na6PDcfb0+CQu5y1qeamu0tGdZ191jX7un1aJB4GYflWx93lbM8opNayy2Gtg39ftebpgNNKV47tV79pmqP7NYL8vblyTF9W786m7HytXV9rzYFcxoT3pH9gN7u+jtZ1hPXwZcHYfqxMOklJRY1DXrNuhQ5B90O6Eh1wWulHmyk5uKnplrgIyquNzQ7VtoWckkr2njzLJSP13Y1mW3fNGExFtZEPHbTJYN0Xs4UTw3U/pAvRM6HawJn7V4wfEMiIvj14e1MGJZXVdtkrZ+0Bc3PaZbo5TbOxEf16MD0qmHd/OM7PZgxqMDHUHjam5dPdx5M/XjPa7q+ltZ7+SbgJEeGiob3JKqrgL+sOs/iftl+jak1qLtFhAQwOcY0NqrTO5a6Zg8krPc83+0/b9XVMJsXGQ/lcFBOig42Lcbmfhoj0EpHVIlIuIlkisqiFvEtFpEZEyqweg62OjxORZBGpsPw7zjHvwj58vT0AMCnbT8gsLDvPzuNFXKqb0zQ7mTU0hKjQAP664QivJRyx28ix/dklnDl3nouH6cmersblAg7wGlANhAGLgWUi0tLWgf9RSgVYPTIARMQb+AL4AAgC/g18YUl3SzOHhuDlYV5Gx2Cw7V45G9LyMCk9Ok2zH4NBmDfcvDmbve7SAb5Lz8cg5o3gNNfiUgFHRPyB64AnlVJlSqktwJfAre0oLh5zH9WrSqnzSqm/YR60MsdW9XU088CFKfTy96Z3gA9jwnvarOw1B3IZ0KsbI/rqSXKa/fh62e8uvc7G9DwmDAyil7/bfrfstFxt0EA0UKuUOmyVlgLMauGcBSJSBOQA/1BKLbOkjwT2qbp10s32WdLXNC5ERJYASwBCQkJITExs95uwt9tihFd3V/GHD75jboRXh8urqFFsOlzBvAhPvv/++/r0srIyl74OjqKvg+2ugf85IwYxBxyDgM/ZLBITT3W8ghbFVSYOZFdyfbSXXX5m+rPQsWvgagEnAGg8nb4E6N5M/pXAW0AeMAX4TETOKqU+spRV0tqylFJvWcoiJiZGxcfHt6f+DjFLKXaW7OCbrFIevXE6PXw7FnS+2JuNUe3l7ssmNxj5lpiYiCtfB0fR18F21yAeCB2UzYMf7+WmyRHcdc2oDpdp7cMdJ4D9LLliKtFhzf3ZaD/9WejYNXBok5qIJIqIauaxBSgDGrfp9ACaXEdGKXVQKXVaKWVUSv0A/BW43nK4TWW5ExHhd5cP52xlDa8nHOtweWsO5BLa3YfxTtrOWutarhrXn0mRQWw9WkDDBoiO+y4tj/CgbgwN1SMtXZFDA45SKl4pJc08ZgCHAU8RGWp12lggtbUvgbmfBss5Y0TEerOaMW0oy6WN6t+Tn4zvz/KtxzlV3P41qiqrjSQeOsMlI/vYbV8fTWts4cQBZBSUsyvTdoMGKquNbDlawNzhYTT8tddchUsNGlBKlQOrgGdExF9EpgNXA+83lV9ErhaRIDGbDDyIeWQaQCJgBB4UER8RecCSvtGub8KBHp4fgwAvrz3U7jI2HTlDZY2RS/XoNM2BrhjTlwAfTz7eZbuVB7ZlFHC+1sQcPRzaZblUwLG4D+gG5AMfAfcqpVIBRGSmiJRZ5b0JOIq5mew94M9KqX8DKKWqgWuA24CzwJ3ANZb0TqFfYDfumjmIz/eeZt+ps+0qY82BXAL9vJg8qJeNa6dpzfPz9uSqcf347/4cSqtss77ahrR8/L09mDJYf5ZdlcsFHKVUkVLqGqWUv1JqoFLqQ6tjm5VSAVbPb1ZKBVvm3wyzDH22LmuPUipWKdVNKTVBKbXHke/FEX4+awjB/t48901am9vDd2QU8s3+HCYMCMTLw+U+Clond+PEAVTVmPhyb8dXHlBKsTEtn5lDQ/Dx9LBB7TR70H9l3Fx3Xy9+NS+aHceLeGhlSqsn0iVnFXPLOzuorjWx+WiB3i9Ec7gx4T0Z1qc7/2liG/e2Sj1dSm5pFXP0VtIuTQecTiAmLAABVu/J5sY3t/HD0YIW8x88XcpvVu6lxmi+IzKZlN4vRHM4EeHGSQPYn11C6unGMxjaZmN6PiIwW68u4NJ0wOkEdmUWUzcop9ak+Om7u3hxTTr5pVUN8uWUVPLwJylc8ffNnDl3Hk+DOGV/H02r85Px/fH2NLCyg3c536XnMzY8kJDuPjaqmWYPrjbxU2uHuMHBeHsaqKk14eFhYNyAQJZ9f4y3N2dw1dj+TIwM4ou92ezOKgaEu2cO5v74KI6eKWN7RiFxg4P1fiGaUwT6eXPpyD6s3pPNby8fXr/0TVvkn6si5eRZfjMv2g411GxJB5xOoG5zOOvgkVVYzvItx/lo50k+221eOsQg8Nqi8Vw2um/9eTrQaM5246QBfJlymrWpuVw9rn+bz09MPwPAxcPDbF01zcZ0wOkkGgePiGB//nD1KHp08+IfjuIpwgAACv5JREFUG4/Wz4jNKCh3Wh01rSlTBwczoFc3Pt55sl0B59Pkk3T38aSy2r7br2sdp/twOrn4mFB8vAy6r0ZzWQaDsDB2ANsyCskqbNsXom3HCtiZWcy587UsfmeHHm3p4nTA6eTqmtsemh+j93bXXNb1E8MR4LHP9rUpaLy87n8Ly9truwPNdnTA6QJiI4K4f3aUDjaayzp9tgoR2J5R1OqN2T7ccYLkrGI89GhLt6H7cDRNc7rtGYXULZRRVWNi85EzLX5B+uFYAU99cYBZ0SHcP3sIuzKL9WhLN6ADjqZpThc3OBgfLwPna0wo4Ot9OdwaF0FwwI/n1RwvKOfeD3YT2dufvy8aTw9fLyYP0nc27kA3qWma5nR1fY0PXxLDY5fGcLKoguvf2MaJwoZbb5RU1vCzf+/CIPDO7RM7vPmg5lg64Gia5hLq+hrvjY/iw7unUFxRzbXLtrL/lHnZm1qjiQc+3M3JogreuCWWiGB/J9dYaysdcDRNczmxEb349OfT8PH04Ka3tvHOlgwWvrmNzUcKeO6a0UzRgwPckg44mqa5pKjQAFbdN43eAT788es0dp84i4dBGKK3j3ZbOuBomuaywnr4cvX4fv9LUHplc3emA46maS5tVnQovnq1jE7B5QKOiPQSkdUiUi4iWSKyqIW834pImdWjWkT2Wx3PFJFKq+PrHPMuNE2zFb1aRufhivNwXgOqgTBgHPCNiKQopVIbZ1RKXWb9XEQSgY2Nsi1QSm2wU101TXMAvbJ55+BSdzgi4g9cBzyplCpTSm0BvgRubcW5kcBM4D171lHTNE1rH5cKOEA0UKuUOmyVlgKMbMW5twGblVKZjdJXiMgZEVknImNtVE9N0zStjVytSS0AKG2UVgJ0b8W5twHPNkpbDOzGvBXML4G1IjJMKXW28ckisgRYAhASEkJiYmLbat4JlZWV6euAvg6gr0EdfR06eA2UUg57AImAauaxBRgPVDQ65zfAVxcodwZQBgRcIF865j6dFusZHR2tNKUSEhKcXQWXoK+DvgZ19HVo+RoASaqFv60OvcNRSsW3dNzSh+MpIkOVUkcsyWOBHw0YaOR2YJVSquxCVcB8t6NpmqY5mEv14SilyoFVwDMi4i8i04GrgfebO0dEugELgXcbpQ8Ukeki4i0iviLyCNAb2Gq3N6BpmqY1y6UCjsV98P/t3W+IXFcdxvHvY7cmNZsYktYILWZJbWqzJTWxIqjBQCOxfSFq3pSNYqAarAbEvmlBJVsNLYLgC2mrwS2tiaRKG1OE0hdCI6YQadDUum1MidBojUpj/mw3zfqHny/OGTOOu9vsbObcmb3PBy7svWcy/O6T4f5m7p49wxXA34A9wJ2Rp0RLWiep9VPMJ4DTwDMtxxcCDwGngFeBjwG3RoT/TNnMrALdNmmAiPg7qYlMNvZL0sSC5mN7SI2p9bGjwOpO1GhmZjPXjZ9wzMxsDnLDMTOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Mi3HDMzKwINxwzMyvCDcfMzIpwwzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Miuq7hSNom6ZCkCUmPXMTjvyLpL5LOSnpY0rymsQFJz0g6J+mIpA0dLd7MzKbUdQ0H+DOwA3j4zR4oaSNwD3ALsBxYAdzb9JA9wG+ApcBXgcclXXWpCzYzszfXdQ0nIvZGxD7g5EU8/LPASESMRsQp4JvAFgBJK4G1wPaIeCMingBeADZ1pnIzM5tOX9UFzNIg8GTT/vPAMklL89gfImKsZXxwsieStBXYmncnJP2uA/X2miuB16ouogs4B2fQ4Bymz2D5dP+w1xtOP3Cmab/x88JJxhrjV0/2RBGxE9gJIOlQRNx8aUvtPc4hcQ7OoME5zC6DorfUJO2XFFNsB9p4yteBRU37jZ/HJhlrjI9hZmbFFW04EbE+IjTF9uE2nnIUuKlp/ybgrxFxMo+tkLSwZXy0/TMwM7N2dd2kAUl9kuYDlwGXSZovaapbfz8E7pC0StJi4GvAIwARcRQ4DGzPz/FJYDXwxEWUsXO25zFHOIfEOTiDBucwiwwUEZeykFmTNAxsbzl8b0QMS3oX8CKwKiKO58ffBdwNXEFqJl+IiIk8NkBqQB8AjgNfioifd/4szMysVdc1HDMzm5u67paamZnNTW44ZmZWhBtOE0lLJP1U0rikVyQNVV1TCdOtXyfplrwO3bm8Lt20f9jVqyTNkzSS/9/HJB2WdGvTeF1y2C3pRF6b8KikzzWN1SKDZpKuk3Re0u6mY0P5dTIuaZ+kJVXW2En5T1nOS3o9b79vGptxDm44/+sB4B/AMmAz8JCkSVcmmGMmXb9O0pXAXuDrwBLgEPDj4tWV0Qf8EfgI8HbSjMef5AVg65TD/cBARCwCPg7skPS+mmXQ7AHgucZOvh58H/gM6TpxDniwmtKK2RYR/Xm7HtrPwZMGMkkLgFPAjXlKNZJ2Aa9GxD2VFleIpB3ANRGxJe9vBbZExAfz/gLSkhZrIuJIZYUWIum3pMVgl1LDHCRdD+wHvgwspmYZSLod+BRpZuy7I+LTku4jNeSh/JhrgZeApS3LaM0JkvYDuyPiBy3H28rBn3AuWAn8q9FssinXXquJQVIGAETEOHCMGmQiaRnpNTFKzXKQ9KCkc8AR4ATwFPXLYBHwDeCulqHWHI6R7oqsLFddcfdLek3Ss5LW52Nt5eCGc0E/cLbl2BnSumx1NdV6dHM6E0mXAz8CHs3v3muVQ0R8kXRu60i30SaoWQakledHIuJPLcfrlsPdpK99uZr0B58/y59m2srBDecCr732/2qXiaS3ALtI79a25cO1yyEi/h0RB4BrgDupUQaS3gtsAL4zyXBtcgCIiF9FxFhETETEo8CzwG20mUOvrxZ9KR0F+iRdFxEv52N1X3ttlPSdQ8B/79tfyxzNRJKAEdIvQW+LiH/moVrl0KKPC+dalwzWAwPA8fSSoJ+0zNYq4Gma1m+UtAKYR7p+1EEAomUdy4vOISK85Q14jPQtoQuAD5E+Ig5WXVeB8+4D5pNmKO3KP/cBV+UMNuVj3wIOVl1vB3P4HnAQ6G85XoscgHcAt5MvsMBGYJw0W60WGeQc3ga8s2n7NvB4zmCQdOt9Xb5O7AYeq7rmDuWwOL8GGteDzfn1sLLdHCo/qW7aSNM99+VQjwNDVddU6LyH8zuX5m04j20g/fL4DdKMpYGq6+1QBsvzeZ8n3S5obJvrkkO+oP4COJ0vJi8An28an/MZTJHLMGmmVmN/KF8fxklfALmk6ho7+Hp4jnSb7HR+M/bR2eTgadFmZlaEJw2YmVkRbjhmZlaEG46ZmRXhhmNmZkW44ZiZWRFuOGZmVoQbjpmZFeGGY9YjJC2SNCzphqprMWuHG45Z77gZ2A5cXnUhZu1wwzHrHWtIXxXwYtWFmLXDS9uY9QBJLwHvaTm8NyI2VVGPWTvccMx6gKT3k1YzHwXuy4dPRMQr1VVlNjP+Phyz3vA86cvQvhsRB6suxqwd/h2OWW8YBN4K/LrqQsza5YZj1hvWkr6v53DVhZi1yw3HrDesAY5FxNmqCzFrlxuOWW9YhadDW4/zpAGz3nAaWCtpI3AGeDkiTlZck9mMeFq0WQ+QdCMwAqwG5gPrIuJAtVWZzYwbjpmZFeHf4ZiZWRFuOGZmVoQbjpmZFeGGY2ZmRbjhmJlZEW44ZmZWhBuOmZkV4YZjZmZF/Ad//c2B2lK/QwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "y_pred = model.predict(X_valid)\n", "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n", "plt.show()" ] }, { "cell_type": "markdown", "source": [ "There are many other models to forecast time series, such as weighted moving average models or autoregressive integrated moving average (ARIMA) models. Some of them require you to first remove the trend and seasonality. Once the model is\n", "trained and starts making predictions, you would have to add them back. When using RNNs, it is generally not necessary to do all this, but it may improve performance in some cases, since the model will not have to learn the trend or the seasonality. " ], "metadata": { "id": "AqHSj9gsDGCv" } }, { "cell_type": "markdown", "metadata": { "id": "xWFI7MMA5jk8" }, "source": [ "## Deep RNNs" ] }, { "cell_type": "markdown", "source": [ "Implementing a deep RNN with `tf.keras` is quite simple: just stack recurrent layers. In this example, we use three SimpleRNN layers. Make sure to set `return_sequences=True` for all recurrent layers (except the last one, if you only care about the last output). **If you don’t, they will output a 2D array (containing only the output of the last time step)** instead of a 3D array (containing outputs for all time steps), and the next recurrent layer will complain that you are not feeding it sequences in the expected 3D format" ], "metadata": { "id": "5Za9vyD9DRtu" } }, { "cell_type": "code", "execution_count": 19, "metadata": { "id": "S63hdy8B5jk8", "outputId": "7b254938-a536-4839-e0da-25876725fb38", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_2\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " simple_rnn_1 (SimpleRNN) (None, None, 20) 440 \n", " \n", " simple_rnn_2 (SimpleRNN) (None, None, 20) 820 \n", " \n", " simple_rnn_3 (SimpleRNN) (None, 1) 22 \n", " \n", "=================================================================\n", "Total params: 1,282\n", "Trainable params: 1,282\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "# By default, recurrent layers in Keras only return the final output. \n", "# To make them return one output per time step, you must set return_sequences=True\n", "# number of parameters https://d2l.ai/chapter_recurrent-neural-networks/rnn.html#recurrent-neural-networks-with-hidden-states\n", "model = keras.models.Sequential([\n", " keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]), #1*20+20+20*20\n", " keras.layers.SimpleRNN(20, return_sequences=True), #20*20++20+20*20\n", " keras.layers.SimpleRNN(1) #20*1+1+1*1\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "code", "source": [ "model.compile(loss=\"mse\", optimizer=\"adam\")\n", "history = model.fit(X_train, y_train, epochs=20,\n", " validation_data=(X_valid, y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SfAv-77vDlcm", "outputId": "7138b70e-7342-4ca4-8549-28f58f537b8d" }, "execution_count": 20, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 41s 178ms/step - loss: 0.0492 - val_loss: 0.0090\n", "Epoch 2/20\n", "219/219 [==============================] - 38s 173ms/step - loss: 0.0070 - val_loss: 0.0065\n", "Epoch 3/20\n", "219/219 [==============================] - 47s 217ms/step - loss: 0.0053 - val_loss: 0.0045\n", "Epoch 4/20\n", "219/219 [==============================] - 62s 282ms/step - loss: 0.0045 - val_loss: 0.0040\n", "Epoch 5/20\n", "219/219 [==============================] - 54s 246ms/step - loss: 0.0042 - val_loss: 0.0040\n", "Epoch 6/20\n", "219/219 [==============================] - 49s 226ms/step - loss: 0.0038 - val_loss: 0.0036\n", "Epoch 7/20\n", "219/219 [==============================] - 37s 171ms/step - loss: 0.0038 - val_loss: 0.0040\n", "Epoch 8/20\n", "219/219 [==============================] - 38s 172ms/step - loss: 0.0037 - val_loss: 0.0033\n", "Epoch 9/20\n", "219/219 [==============================] - 54s 245ms/step - loss: 0.0036 - val_loss: 0.0032\n", "Epoch 10/20\n", "219/219 [==============================] - 37s 169ms/step - loss: 0.0035 - val_loss: 0.0031\n", "Epoch 11/20\n", "219/219 [==============================] - 37s 170ms/step - loss: 0.0034 - val_loss: 0.0030\n", "Epoch 12/20\n", "219/219 [==============================] - 37s 168ms/step - loss: 0.0033 - val_loss: 0.0031\n", "Epoch 13/20\n", "219/219 [==============================] - 37s 170ms/step - loss: 0.0034 - val_loss: 0.0031\n", "Epoch 14/20\n", "219/219 [==============================] - 38s 173ms/step - loss: 0.0033 - val_loss: 0.0032\n", "Epoch 15/20\n", "219/219 [==============================] - 38s 174ms/step - loss: 0.0034 - val_loss: 0.0033\n", "Epoch 16/20\n", "219/219 [==============================] - 38s 172ms/step - loss: 0.0035 - val_loss: 0.0030\n", "Epoch 17/20\n", "219/219 [==============================] - 38s 173ms/step - loss: 0.0033 - val_loss: 0.0029\n", "Epoch 18/20\n", "219/219 [==============================] - 37s 170ms/step - loss: 0.0033 - val_loss: 0.0030\n", "Epoch 19/20\n", "219/219 [==============================] - 37s 170ms/step - loss: 0.0032 - val_loss: 0.0029\n", "Epoch 20/20\n", "219/219 [==============================] - 37s 169ms/step - loss: 0.0032 - val_loss: 0.0029\n" ] } ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "42_p59jH5jk8", "outputId": "4bd43337-53b7-451b-87a7-4acf6cc28e3a", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 3s 47ms/step - loss: 0.0029\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "0.002910560928285122" ] }, "metadata": {}, "execution_count": 21 } ], "source": [ "model.evaluate(X_valid, y_valid)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "fsQp1KDq5jk9", "outputId": "4d353299-c25f-4a96-da4b-f6d63ae212d6", "colab": { "base_uri": "https://localhost:8080/", "height": 291 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1fn48c8zyWQhIYGwJCxh0bJIkL1iFASLoFZBK+jP4sZXKlbrQhWhtnUp+tVS4au2oqKoqPBVtMpXW6kiCGURUbSoLEqrgGxhFUhC1pnn98edhEkcIMvcmYE879frvmbm3DP3PjOZzDPn3HvPEVXFGGOMCTdPtAMwxhhzcrIEY4wxxhWWYIwxxrjCEowxxhhXWIIxxhjjCkswxhhjXGEJxhhjjCsimmBEJENE5olIoYhsEZHRR6knIjJFRPYFlikiIkHrNbCNgsAyM3KvwhhjTE3ER3h/04FSIBPoBbwjIp+r6rpq9cYBlwI9AQXeBzYBTwfV6amq/3E/ZGOMMXURsRaMiKQAI4F7VLVAVZcDbwPXhKh+HTBNVbep6nZgGjAmUrEaY4ypv0i2YDoD5aq6Majsc2BQiLo5gXXB9XKq1VkqIh7gQ+AOVd0caqciMg6nRURycnLf7OzsukUf4Pf78Xhi49BVLMUCsRWPxRJaLMUCsRWPxRLaxo0b96pqizo9WVUjsgADgbxqZTcAS0LU9QFdgx53wukqk8Djc4AEoAnwBLAWiD9eDH379tX6Wrx4cb23ES6xFItqbMVjsYQWS7GoxlY8FktowGqt4/d+JFNkAZBWrSwNyK9B3TSgIPBiUdWlqlqqqgeA24GOwGnhD9kYY0xdRTLBbATiRaRTUFlPoPoBfgJlPWtQr4ICcoz1xhhjIixiCUZVC4E3gckikiIiZwOXAC+HqP4ScIeItBGR1sCdwCwAEckRkV4iEiciqTgnAGwHNkTidRhjjKmZSB9FuhlIBnYDrwA3qeo6ERkoIgVB9WYAfwO+xDm+8k6gDJxTnOcCh4BvgQ7AxapaFpFXYIwxpkYieh2Mqu7Hub6levkyIDXosQITA0v1uh8AXVwM0xhjTBhE+kJLY0w9HDp0iN27d1NWVrcGe3p6Ohs2xE5vcizF01BjSUlJoW3btq6cFm0JxpgTxKFDh9i1axdt2rQhOTmZoNGTaiw/P5/GjRu7EF3dxFI8DTEWv9/P9u3b2bt3Ly1btgz79mPjSh5jzHHt3r2bNm3a0KhRozolF2Oq83g8ZGZmcvDgQXe278pWjTFhV1ZWRnJycrTDMCcZr9dLeXm5K9u2BGPMCcRaLibc3PxMWYIxxhjjCkswxhhjXGEJxhhzwhkzZgwXX3xxrZ4zePBgbrnlFpciOuL++++ne/furu/nRGCnKRtjXHO8/v3Ro0czZ86cWm/38ccfrxhpvcbefPNNvF5vrfdl6s4SjDHGNTt37qy8//e//50bbrihSln1s5fKyspqlATS09NrHUtGRkatn2Pqx7rIjGlgVq6Ehx92bt2WlZVVuTRp0qRKWXFxMdnZ2bzyyiv85Cc/ITk5mRkzZrBv3z5+/vOf07ZtW5KTk8nJyeGFF16ost3qXWSDBw/m5ptv5re//S3NmzenZcuWTJgwAb/fX6VOcBdZhw4dePDBB7nxxhtJS0uja9euPPLII1X2s3HjRgYNGkRSUhJdunRh/vz5pKamMmvWrBq/B36/nwceeIDs7GwSExM5/fTTeeutt6rUmTx5Mu3btycxMZGsrCzGjRtXuW7p0qWceeaZpKamkp6ezhlnnMHatWtrvP9oshaMMSeo8eNhzZraPef775NZuxb8fvB4oEcPqE1joFcveOyx2u3zeO6++26mTp3Kc889h9frpbi4mD59+jBp0iTS0tJYuHAhN954I+3atWPIkCFH3c6cOXO4/fbb+fDDD1mzZg2jR4+mb9++/PznPz/qcx599FH+8Ic/cNdddzFv3jwmTpzIgAEDyM3Nxe/387Of/YysrCw++ugjioqKGD9+PCUlJbV6fY8//jiPPPIITz/9NP369WP27NlcdtllfPrpp/Tq1Ys33niDqVOn8sorr3D66aeze/dulixZAjgtvEsuuYSxY8cyZ84cysrK+Oyzz4iLi6tVDNFiCcaYBuTgQaHiR73fDwcP1i7BuOHWW29l1KhRVcruuuuuyvvjxo3jgw8+4JVXXjlmgunWrRuTJ08GoHPnzjz77LMsWrTomAlm2LBhla2aX/7ylzzzzDMsWrSI3Nxc3n//fb7++msWLFhAmzZtACchnX322bV6fVOnTmXChAmMHj0acForS5cuZerUqcyePZstW7bQqlUrhg0bhtfrpV27dnTp4ozne+jQIQ4cOMDw4cM59dRTAejatWut9h9NlmCMOUHVpSWxcGExI0akUFoKCQkwZw7k5oY/ttro169flcc+n48//vGPzJ07l+3bt1NSUkJpaSmDBw8+5nZ69OhR5XHr1q3ZvXt3nZ/z1Vdf0bp168rkAvDjH/+4VoNCHjp0iB07dvwgKQ0YMID58+cDcPnll/P444/TsWNHzj//fC644ALOPfdcGjduTEZGBmPGjOH8889nyJAhDBkyhFGjRtGuXbsaxxBNdgzGmAakf38/ixbBAw/AokXRTy7gjOYbbOrUqUybNo277rqLRYsWsWbNGi699FJKS0uPuZ3qJweISJVjMOF6TrhUnGGXnZ3N119/zYwZM0hLS+POO+/knHPOobCwEIAXXniBVatWcc455/D222/TpUsX3nvvvYjEWF+WYIxpYHJz4e67YyO5hLJ8+XKGDx/ONddcQ69evTj11FPZuHFjxOPo2rUrO3bsYMeOHZVlq1evrlUCSktLo3Xr1qxYsaJK+fLly+nWrVvl46SkJC666CIeffRRPvnkEzZs2FDlOT179mTSpEksWbKEwYMH8+KLL9bjlUWOdZEZY2JK586dmTt3LsuXL6d58+b85S9/YdOmTfTu3TuicQwdOpQuXbpw3XXXMXXqVIqKirjjjjuIj4+v1fhdd911F/feey+dOnWib9++zJ49m2XLlvHZZ58BMGvWLMrLy+nfvz+pqanMnTsXr9dLp06d2LRpEzNmzGDEiBG0adOGb7/9li+++IKbbrrJrZcdVpZgjDEx5fe//z2bNm3iwgsvJDk5mTFjxnDVVVexfv36iMbh8XiYN28ev/jFLzjjjDPo0KED06ZN47LLLiMpKanG27ntttvIz89n4sSJ7Nq1iy5duvDGG2/Qs2dPAJo0acKUKVOYMGECZWVldOvWjdmzZ9OxY0d27drFxo0bufzyy9m7dy+ZmZlcddVVTJo0ya2XHV6q2mCWvn37an0tXry43tsIl1iKRTW24jkZY1m/fn29t3Ho0KEwRBI+sRRPTWJZs2aNArp69eqoxxJOx/psAau1jt+51oIxxpijmDdvHikpKXTq1InNmzdzxx130LNnT/r06RPt0E4IlmCMMeYo8vPzmTRpElu3bqVp06YMHjyYRx991OblqSFLMMYYcxTXXnst1157bbTDOGHZacrGGGNcYQnGGGOMKyzBGGOMcYUlGGOMMa6wBGOMMcYVlmCMMca4whKMMSbm3X///XTv3v2oj0O55ZZbjjvEf1327Zbqs3SeDCzBGGNcM2LEiKNOErZhwwbS0tJYsGBBrbc7YcIE/vnPf9Y3vCq2bNmCiLB69WrX99VQWIIxxrhm7NixLF68mM2bN/9g3XPPPUe7du0477zzar3d1NRUmjVrFoYIY2tfJxtLMMY0NCtXwsMPO7cuu+iii8jMzOSFF16oUl5WVsbLL7/M1VdfjaoyduxYOnbsSHJyMp06deJPf/rTMeddqd5t5fP5mDBhAk2bNqVp06aMHz8en89X5TnvvvsuAwcOpGnTpmRkZHD++eezYcOGyvWnn3464MxaKSKV3WvV9+X3+3nggQfIzs4mMTGR008/nbfeeqty/ebNmxER3njjDYYOHUqjRo3o1q0b77//fq3eu5KSEsaPH09mZiZJSUmceeaZLF++vMp7eNttt9G6dWsSExPJzs7mN7/5TeX6N998kx49epCcnExGRgaDBg1i165dtYqhvmyoGGNOVOPHw5o1tXpK8vffw9q14PeDxwM9ekB6es030KtXreZqjo+P57rrrmPWrFncd999ldMN/+1vf2Pv3r1cffXV+P1+2rRpw2uvvUaLFi34+OOPGTduHM2aNWPs2LE12s+0adN49tlnefbZZ+nRowfTp09nzpw5VQalLCwsZPz48fTo0YOioiIefPBBhg8fzvr160lISGDx4sWce+65vPvuu/Ts2ZOEhISQ+3r88cd55JFHePrpp+nXrx+zZ8/msssu49NPP6VXr16V9X73u9/xyCOP8OSTT/Lggw9y5ZVXsmXLFlJTU2v0miZOnMhrr73G888/zymnnML//M//cMEFF/Dvf/+bVq1a8ec//5l58+bx6quv0qFDB7Zt28bXX38NQF5eHldeeSUPP/wwI0eOpKCggI8++qhG+w0nSzDGNCBy8KCTXMC5PXiwdgmmDsaOHcuUKVNYuHAhw4YNA5zusWHDhtG2bVu8Xi+TJ0+urN+hQwc+++wzXnnllRonmMcee4yJEydyxRVXAE4SqD6t8MiRI6s8fuGFF0hLS+Pjjz9mwIABNG/eHIBmzZqRlZV11H1NnTqVCRMmMHr0aAAmT57M0qVLmTp1KrNnz66s9+tf/5rhw4cD8NBDD/HSSy+xZs0aBgwYcNzXU1hYyFNPPcXMmTO56KKLAHj66af54IMPmD59Og8++CBbtmyhc+fODBw4EBGhXbt2nHXWWQDs2LGDsrIyRo0aRfv27QEicqJCdRFNMCKSATwHDAP2Aner6v+GqCfAH4FfBIpmAr8JzE0QXO9a4EXgBlWd6WbsxsScWrQkKhQvXEjKiBFQWgoJCTBnjutzJ3fq1IlBgwbx/PPPM2zYMHbs2MF7773Hq6++Wlnn6aefZubMmWzZsoWioiLKysoqvxiP5+DBg+zcuZPcoNfh8Xjo378/W7durSz75ptvuOeee1i1ahV79uzB7/fj9/v57rvvavxaDh06xI4dOzj77LOrlA8YMID58+dXKevRo0fl/datWwOwe/fuGu1n06ZNlJWVVdlPXFwcubm5lROvjRkzhqFDh9K5c2eGDRvGT3/6Uy688EI8Hg89e/bkvPPOo3v37gwbNozzzjuPUaNG0aJFixq/1nCI9DGY6UApkAlcBTwlIjkh6o0DLgV6Aj2A4cCNwRVEpCnwW2CdmwEbczLx9+8PixbBAw84ty4nlwpjx47l//7v/9i/fz+zZs0iIyODSy65BIC5c+cyfvx4xowZw3vvvceaNWu4+eabKS0tDWsMF198MXv27GHGjBmsWrWKf/3rX8THx4dtP9WH8Pd6vT9Yd6zjSrXdT58+fdi8eTMPP/wwfr+f6667jqFDh+L3+4mLi2PBggUsWLCAHj168Nxzz9GpUyc+//zzeu+/NiKWYEQkBRgJ3KOqBaq6HHgbuCZE9euAaaq6TVW3A9OAMdXqPAz8GaclZIypqdxcuPvuiCUXgFGjRpGUlMTs2bN5/vnnufbaayu/gJcvX07//v255ZZb6NOnDz/60Y/45ptvarzt9PR0WrVqVeUYg6ry8ccfVz7et28fX331Fb/97W8577zzOO2008jPz6e8vLyyTsUxl+onBwRLS0ujdevWrFixokr58uXL6datW41jPp6OHTuSkJBQZT8+n4+VK1dW2U/jxo0ZNWoUTz31FO+88w4ffPAB//nPfwAnEeXm5nLffffxySef0Lp1a+bOnRu2GGsikl1knYFyVd0YVPY5MChE3ZzAuuB6lS0dETkD6AfcDFxxrJ2KyDicFhGZmZksWbKkLrFXKigoqPc2wiWWYoHYiudkjCU9PZ38/Px6bcPn89V7G3U1atQo7rvvPg4cOMCVV15Jfn4+Pp+Pdu3aMWvWLN544w1OOeUU3njjDf75z3/SpEmTylhLSkrw+/1HffzLX/6SP/3pT2RnZ5OTk8Ozzz7Lzp07admyJfn5+cTHx9OsWTOefPJJmjZtys6dO/n9739PfHw8xcXF5Ofnk5GRQXJyMm+//TbNmzcnMTGR9PT0H+zr1ltv5aGHHqJt27b06tWLuXPnsmzZMpYtW0Z+fj4FBQWAcxyl+ntdVFR01Pe/rKyM8vJy8vPzSUpKYuzYsUycOJFGjRrRoUMHpk+fzq5du7j22mvJz8/niSeeIDMzkx49ehAfH8+sWbNIS0sjPT2dRYsWsWTJEoYMGULLli354osv2Lp1Kx07dgy5/+LiYnf+X+o613JtF2AgkFet7AZgSYi6PqBr0ONOgAICxAGrgTMD65YAv6hJDH379q3R/NTHcjLO9R4usRTPyRjLseZNr6lIz/Ue7NNPP1VAzzrrrCrxlJSU6PXXX69NmjTR9PR0vf766/UPf/iDtm/fvrLefffdpzk5OUd9XFZWpuPHj9f09HRNT0/XW265RX/5y1/qoEGDKussWrRIc3JyNDExUXNycvTdd9/VlJQUfeGFFypjefbZZzU7O1s9Hk/lc6vvy+fz6eTJk7Vt27bq9Xq1e/fuOm/evMr1mzZtUkA/+eSTKq8f0Ndff/2o7891112nF110UWUsxcXFevvtt2vLli01ISFB+/fvr8uWLaus/8wzz2jv3r01NTVVGzdurOecc46uWLFCVZ3PygUXXFD53FNPPVWnTJly1H0f67MFrNa6fu/X9Ym13hH0Bg5XK7sT+FuIugeBM4Ie9wXyA/dvBZ4PWmcJJkbEUjwnYywneoIJJZbiacixuJVgInmQfyMQLyKdgsp6Evog/brAulD1hgA/E5E8EckDzgKmicgTLsRsjDGmjiJ2DEZVC0XkTWCyiPwC6AVcgpMgqnsJuENE5uN0jd0J/CWwbgyQFFT3TeCvOKc/G2OMiRGRvtDyZuB5YDewD7hJVdeJyEDgH6pacYnrDOAU4MvA45mBMlT1QPAGRaQUOKSqByMQvzHGmBqKaIJR1f0417dUL18GpAY9VmBiYDneNgeHMURjjDFhYoNdGnMC0aqDWRhTb25+pizBGHOC8Hq9FBUVRTsMc5IpKysjPt6dzixLMMacIFq2bMn27ds5fPiwtWRMWPj9fnbt2kW6SwOe2mjKxpwg0tLSgCMj5dZFcXExSUlJx68YIbEUT0ONJSUlpXIk6XCzBGPMCSQtLa0y0dTFkiVL6N27dxgjqp9YisdiCT/rIjPGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcYUlGGOMMa6wBGOMMcYVlmCMMca4whKMMcYYV1iCMcYY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY4wxrrAEY4wxxhWWYIwxxrjCEowxxhhXWIIxxhjjCkswxhhjXGEJxhhjjCsswRhjjHGFJRhjjDGusARjjDHGFZZgjDHGuMISjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcUVEE4yIZIjIPBEpFJEtIjL6KPVERKaIyL7AMkVEJLCuuYisCJQfEJGVInJ2JF+HMcaY44uP8P6mA6VAJtALeEdEPlfVddXqjQMuBXoCCrwPbAKeBgqA64F/B9ZdAvxNRFqqanlEXoUxxpjjilgLRkRSgJHAPapaoKrLgbeBa0JUvw6YpqrbVHU7MA0YA6Cqxar6tar6AQF8QFMgIwIvwxhjTA2JqkZmRyK9gRWq2iiobAIwSFWHV6t7EBimqqsCj/sBi1W1cVCdL4CugBeYqao3HGW/43BaRGRmZvZ99dVX6/U6CgoKSE1Nrdc2wiWWYoHYisdiCS2WYoHYisdiCe3cc8/9VFX71enJqhqRBRgI5FUruwFYEqKuD+ga9LgTTneYVKuXBPwcuK4mMfTt21fra/HixfXeRrjEUiyqsRWPxRJaLMWiGlvxWCyhAau1jt/7kTwGUwCkVStLA/JrUDcNKAi82EqqWgy8IiIbRGSNqn4ezoCNMcbUXSTPItsIxItIp6CynkD1A/wEynrWoF4FL3BKvSM0xhgTNhFLMKpaCLwJTBaRlMCpxZcAL4eo/hJwh4i0EZHWwJ3ALAAROVNEBohIgogki8gknLPSVkXkhRhjjKmRSJ+mfDPwPLAb2AfcpKrrRGQg8A9VrTiqNQOnRfJl4PHMQBlAIvDnwPqyQJ2LVHXH8Xbu94frZRhjjDmeiCYYVd2Pc31L9fJlQGrQYwUmBpbqdf9J1e6zGisrq8uzjDHG1EWDGirGEowxxkROg0ow5XadvzHGREyDSjClpdGOwBhjGo56JxgR8YYjkEiwFowxxkROrRKMiNwmIiODHj8HFInI1yLSJezRhZkdgzHGmMipbQvmNmAPgIicA1wBjAbW4AxIGdMswRhjTOTU9jTlNjjD5gMMB15X1ddE5EtgWVgjc4ElGGOMiZzatmAOAS0D94cCiwL3y3AGnoxplmCMMSZyatuCWQA8KyKfAT8C/hEoz+FIyyZmlZWBzwdxcdGOxBhjTn61bcH8ClgBtABGBa7MB+gDvBLOwNyyd2+0IzDGmIahVi0YVT0E3Bqi/L6wReSyvDzIzIx2FMYYc/Kr7WnK3YJPRxaRoSIyW0TuFpETouNp585oR2CMMQ1DbbvIngd6A4hINvAWkIHTdfZgeENzR15etCMwxpiGobYJpivwWeD+KGCVqv4UuAZn6uKYZy0YY4yJjNommDigYkSvIcD8wP1vcCb9imkej7VgjDEmUmqbYNYCNwUmCBsCvBsobwPE/PlZXq+1YIwxJlJqm2AmATcAS4BXVLVixskRwMdhjMsVlmCMMSZyanua8lIRaQGkqer3QatmAIfDGpkLvF7rIjPGmEip9ZTJquoTkSIR6Q4o8I2qbg57ZC6wFowxxkROba+DiReRR4Dvgc+BL4HvReRPJ8K8MF4vFBZCQUG0IzHGmJNfbY/B/Am4Gvgl0BnoBNyEc5ryw+ENLfy8gRRorRhjjHFfbbvIRgPXq+r8oLJvRGQPMBOYELbIXFCRYPLyoFOn6MZijDEnu9q2YNJxrnmp7hugSf3DcZe1YIwxJnJqm2A+x5nVsrrbA+tiWnALxhhjjLtq20U2EZgvIucBHwXKzgRaAxeGMzA3xMfbmWTGGBMptWrBqOpSnIP7fwVSA8vrwPmEbtnEnMxMSzDGGBMJdbkOZgfwu+AyEekJjAxXUG5q1cq6yIwxJhJqewzmhJeVZS0YY4yJhAaXYKwFY4wxkdEgE8yePVBeHu1IjDHm5FajYzAi8vZxqqSFIZaIyMoCVdi9G1q3jnY0xhhz8qrpQf59NVi/qZ6xRESrVs7tzp2WYIwxxk01SjCq+l9uBxIpWVnOrR2HMcYYdzXIYzBgZ5IZY4zbIppgRCRDROaJSKGIbBGR0UepJyIyRUT2BZYpIiKBdZ1F5C0R2SMi+0XkPRHpUtMYMjOdW0swxhjjrki3YKYDpUAmcBXwlIjkhKg3DrgU6An0AIYDNwbWNQHeBroEtvMx8FZNA0hMhIwM6yIzxhi3RSzBiEgKztX+96hqgaoux0kU14Sofh0wTVW3qep2YBowBkBVP1bV51R1v6qWAY8CXUSkWU1jsYstjTHGfaKqkdmRSG9ghao2CiqbAAxS1eHV6h4EhqnqqsDjfsBiVW0cYruXAk+paquj7HccTouIzMzMvq+++ip33tmTkhIPTzzxr1q/joKCAlJTU2v9PDfEUiwQW/FYLKHFUiwQW/FYLKGde+65n6pqvzo9WVUjsgADgbxqZTcAS0LU9QFdgx53ApRAQgwqbwtsB35ekxj69u2rqqpXX63aoYPWyeLFi+v2RBfEUiyqsRWPxRJaLMWiGlvxWCyhAau1jt/7kTwGU8APL8hMA/JrUDcNKAi8WABEpAWwAHhSVV+pTSBZWc4xmAg13owxpkGKZILZCMSLSPBkxT2BdSHqrgusC1lPRJriJJe3VfW/axtIq1ZQXAwHD9b2mcYYY2oqYglGVQuBN4HJIpIiImcDlwAvh6j+EnCHiLQRkdbAncAsABFJA97DOZ7zm7rEUnGxpR3oN8YY90T6NOWbgWRgN/AKcJOqrhORgSJSEFRvBvA34EtgLfBOoAzgZ8CPgf8SkYKgpV1Ng6i42NJOVTbGGPfUesKx+lDV/TjXt1QvX4YzO2bFY8WZnnliiLovAi/WJw5rwRhjjPsa3FAxYC0YY4yJhAaZYNLTISnJWjDGGOOmBplgRI6cqmyMMcYdDTLBgNNNZi0YY4xxT4NNMNaCMcYYdzXYBGMtGGOMcVeDTTBZWbB/P5SURDsSY4w5OTXYBFNxqvKuXdGNwxhjTlYNNsHYxZbGGOOuBptg7GJLY4xxV4NPMNaCMcYYdzTYBNOypXPBpbVgjDHGHQ02wcTHQ4sW1oIxxhi3NNgEA3axpTHGuKlBJxi72NIYY9zToBNMVpYlGGOMcUuDTjCtWjkXWvr90Y7EGGNOPg0+wZSVOUPGGGOMCa8GnWAqrua3A/3GGBN+DTrB2MWWxhjjngadYKwFY4wx7mnQCcZaMMYY454GnWBSUyElxVowxhjjhgadYMAutjTGGLdYgrEEY4wxrmjwCcbGIzPGGHc0+ARjLRhjjHFHg08wWVlw6BAcPhztSIwx5uTS4BOMTZ1sjDHuaPAJxi62NMYYdzT4BGMXWxpjjDsafIKpaMFYgjHGmPBq8AmmRQuIi7MuMmOMCbeIJhgRyRCReSJSKCJbRGT0UeqJiEwRkX2BZYqISND6Z0TkaxHxi8iY+sTk8UBmprVgjDEm3CLdgpkOlAKZwFXAUyKSE6LeOOBSoCfQAxgO3Bi0/nPgZuCzcARlF1saY0z4RSzBiEgKMBK4R1ULVHU58DZwTYjq1wHTVHWbqm4HpgFjKlaq6nRVXQQUhyM2u9jSGGPCT1Q1MjsS6Q2sUNVGQWUTgEGqOrxa3YPAMFVdFXjcD1isqo2r1VsOzFTVWcfY7zicFhGZmZl9X3311R/UeeSRLqxalcFf/7ryuK+joKCA1NTU49aLhFiKBWIrHosltFiKBWIrHosltHPPPfdTVe1XpyerakQWYCCQV63sBmBJiLo+oGvQ406AEkiIQeXLgWqAcBwAABjnSURBVDE1jaFv374ayu9/r+rxqJaXh1xdxeLFi49fKUJiKRbV2IrHYgktlmJRja14LJbQgNVax+/9SB6DKQDSqpWlAfk1qJsGFARebNhlZYHfD3v3urF1Y4xpmCKZYDYC8SLSKaisJ7AuRN11gXXHq1c7eXmw8ofdYHaxpTHGhF/EEoyqFgJvApNFJEVEzgYuAV4OUf0l4A4RaSMirYE7gVkVK0UkQUSSAAG8IpIkIsd/Ldu3w5AhP0gylmCMMSb8In2a8s1AMrAbeAW4SVXXichAESkIqjcD+BvwJbAWeCdQVmEBUAScBTwTuH9OjSIoKoKnn65SZOORGWNM+MVHcmequh/n+pbq5cuA1KDHCkwMLKG2M7jOQYjASy9BQQE8/ji0bWvDxRhjjAsa1lAxbdrAkiXw0EMwfz6cdho89hjJ3nLS060FY4wx4dSwEkxWFpxzDtx9N6xbBwMGwK9/DWecwdAmn1gLxhhjwqhhJZhgp5zitGJeew3y8pi7pT9XrrgVDh6MdmTGGHNSaLgJBpzjMZdfDhs28H6nX/GzndOdbrPXXoMIjXBgjDEnq4adYCqkp/NCn79wVtzHFKS1gv/3/+CnP4Vvv412ZMYYc8KyBINzWcybb8IqXz9abVnFptsfg+XLISfHOSGgtDTaIRpjzAnHEgzOiWU+n3O/oDieX3x5O7v++RVcdBH87nfQuzcsWxbVGI0x5kRjCQYYPBgSE53Jxzwe+OADyD6zDVcl/pWvp/0dCguds88uvpgOzz0XcrgZY4wxVVmCAXJzYdEiePBBp2ds40a4+Wb4+9+h650XMaj5Or496yr0nXdoP3u2k2wWLIh22MYYE9MswQTk5jqXx+TmQqdO8NhjsG0bPPEE7CpI4dkPc/DjQQDKy2HECLj3Xti/P9qhG2NMTLIEcwyNG8OvfgXr18Oljw6m3JNIGXEUk8j69Fx44AFo3x5++1sb698YY6qxBFMDHg/0H59L4vJFrLtiHE9fsZjc4sV050sWJFyE/vGPaIcOMHEi7NoV7XCNMSYmWIKpjdxcDtx0BePn5rJtG9z0RHdua/Eq3XQdb/guxT91Gr52Hfl4wB2s/puNO2OMadgswdRRcPfZ4++dxqwhs+mqG3i59Ar6rPgz3Ud05OPc29i0bBt+f7SjNcaYyLMEU08eDwwb5pxxNuLOzlwvs+jC18zmanp/9BStzzmV5xJv5rJ+3/GrX8HMmbB6NRQXRztyY4xxV0TngznZjRwJTz4JW0pP5baEmTSZ/Hu6//2P/NeymYz5bCazvxjD5LK7ySKPn8gSNncYjOfsXHr1onJp1szZ1sqVzgWggwc7Z7YZY8yJxhJMGFVcT3MkMXSACU/D1t/BlCmMefZZxnieQxHwK+VbvFx/cB4TZl+AM/szZGdDu3awapUzuoDX65wyPXgwtGgBGRlOq8kYY2KdJZgwy80N0eLIzoYnnkDuvhuuuAL58EMAEvwlzN7/U15OTuZwRlv2JmezxZfNJ/9qx2nl2Wwlm62l2Uy6OZt80gAnuTRrBuelrKR/8Qcs6JrI911zadECmjd3klDFsmULfPEFDBlyjFZQLDWVYikWY0y9WYKJpDZtYOpU5xu/tBTi4uCmm5C4OFK2biVl61bab13IwOKdCFXPDChNTiM/PZt9KdkcLk+g+5b5eNRHed4U/vLxRL4s6sQmdVpBStXb9fcIc5pCRjOhcZrQuDE0ThPal2wk94MH8fjKUa+XPX+ZS8qoC0lpmoDIkX2H43s/5Db8fti3z5mretEimDTJuYg1IQFefx0uvpgqgRhjTiiWYCLth/1oP6gi5eV8+rcdrHtvK2e23krn5K0kbN1Ks8DCV1+BlgOQQCl3Hn7w+Pv9PrAchZSWkHnjpXAj7CODffFZHEjO4vuELDbsz+SAZvGMJ4uFZ2WR2C6TwsZZlDRujscbR1wctN+xkiafv8OL/RLJa9+f1JJ9pBfuoHHBToq/3cG6hTvJ8u9gt+xga+udZBTvIOlAHnG+sh8GU1ICI0ZQltqUwg45FHboRlHHHIpOyWFNWQ4fbcok9yyhf39nDLmkpCOL11uznGSNJWPcZwkmGkL2owWJj6fvz9rR92ftQq9fuRKGDEFLSpCEBHj5ZWfEZ6icKO1fnynXXgvlZUqCV5k1C3r30so6qlC08l8k/WosUlaG3xPH2qG/5lB5CnF78kjYn0dqfh4tD6xigO4khcPgB5YfCcOHh920JJ9UTuVbPPjRT6fhx0M8vh+EvZ+m7NDWfLW9FTs4l520Yget2UkrmrKfxxlPPGX4iGc6N5NScJictevIWfs6bXgGgNOBn5LB+qe6sYgc1gUtu2kJCIMSVjKY5TySksi6tNzK5FORjIqLnTP5/H6nEXnBBdC2rbM+IeHIbfD9UGXffOPMvN2/P5x5JjRq5CzJyc5tXFzNPg7hSnautTRjOZYaVLIfE9FjCeZEFGgFbXr+eU65/vqQ/zW9fwTPZB/5x+pdrYoAjbqdBt06wpIlxA0eTM8Q21m5EroNAW9JAdnePF59fBfdm+dBXh5xeXm02rWLVsuWoV/5nQ45ATl7AMUjRlLeohXlLVvz2c5WXParVhSUJ+H1Ovnw7F7OSQx+/5Hb7z7rQerqJRzsPZizc3Iry78oV+L37WLlzHVsW7CObuqklKu9c0ktO1AZ6+HkDA6mZdNy91pE/fjLJ/N+1q18l3Iahf5kCkuSKDiczJZdSXj8yRSTRJEvme+WJrE5MZlDZckcKk2ioMSLzy+cyUoGs4QlDOYjjv7N9MwzocsTEmBQwkoG+pcxvWkiX2fk/iAJFRY646ZWJLvLLoPWrZ1WWG2W7dud99Xnc7Zz6aXOMbnS0iNLx7yVtPt2EfNaJfJ5o9wq60pL4dAhyMtzfqOIOEm3SZMjSbUisfYoXEnPA0v4d+vBfNcmt8q6xERnGy++6PR2xsXB6NHOMcHSUqdxWlLi3G+/YyXtNi1kXssE1if1QYtL0OISpKSYkkMlHNhVQgIl/J1imqWUkOwpIUFLSNRivFpC5/L1/Kr0UeIox4eXe73/zUrPAPZrBnu1Gd9rE8o1rnIqDoBWrZz3pXHjI0tqqnO7f39HVq4Mva5xY2cQ3FWrnB8Tffs621Ot2W3F/X/9y9nGWWc5/7Ze75ElIcF5vyLZAnc7+Yo2oKmB+/Xrp6tXr67XNpYsWcLgwYPDE1A9RSqW434IAy0qf0kJnsREpwuwWsVw/aKtOHyVkACLFiq5HfOcZkTFsmCBc3ZDfXg8qNfrfBMCIPhaZKLJKfjj4vF74tl3IJ68ffH4iKOceDJaxJPePJ5y4inXeMqIJ77wIB23LUfUh1/i+CRrBDsS2lPkS6TQl8jh8kT25idyoDiREpzFH5+ILz6RYpIoIZFidco7+TbQ17+aT6Ufa+V0UEXUf2TBj6B48OPBT1KCn8aN/CTE+/HGK6f51zJhz2+I0zJ8Es8LHf7A7sY/IiHOR2JcOV6Pj907fezcWo4HH/H4aNvKR2azcvzlPrTMB+XlNC/czMV7X8SDDz9xvJt0CfvJIN5XSpyvlHh/CQmUVi6JOI+TpJREKSGRUhKklEb+AhrrocC76w4/QkF8E3aXZ7CPZuwnA3/TZmgT5/EeXwa7yzPYWdqMJoXbOaVoLUsZyAoGcJhGFJGMnxo2QwNq+qPkWAbErWSwfMCqpJ+wJjm3MvlUJKLSUqf1rOqc9NO1q5MAK1R8pXfPX0nf/CV82ngwaxvn/mAW+IICp7ddxPlhEOLfFlaupO1ZZ23fptq2Lq/FEkwtNcQEUyMrV/LtMVpUYdxN7ZLdG29A9+5Ov1hRUeXt+s+KWf9ZET1+VETndlXXUVzs7GTFiiM/53v3htNOc36Wl5ezf3c5q1aU4/GX4/WU07dnOekpvsr1lJc7Jy8Ej01X0W9W8RP+ZJCcDOnpVfoOC0oTWPefREo0gTJPAn3OTKRpZkLVPsYvv4SPPz7y/g4dCuefX9mP+e/vErn/j4kUlifii0/ij48mktMn8UhfZ2IirF3rTG9eVuZ88/7lL86JNPv2OaOc79vHzvX7WTpvH018+2km++neah9Jh/fDgQPHf22AP96LL7ERBb5k9hc7SaeIZBo1b0R6VjLlCY3wJSTj8ybjLTpI+8/+D/H7UE8c23tcSGmjJsT5SvGUl7J3ewn780rxBhJvi/RSmjQqxVNWiqe8BE95Kd7SQhJL8yv3fzghnbL4RpSLl3LxUiYJFJZ6KSjxUkoCZXjxJnvxpngplwTKPV584iWl/AB99r2PqB8VDx83/ykHEjOrJPNDh+Cgk+MRgX59nWvxKu3aBfPn08/nY7VqnX4HWBeZCY/cXL4rKeEUlzu5j3f4qqL7cPNxkl23QdDtWDuq3lx64okq28oAmgQlu/SaJLuFC49sQ7Wyz+iT5SWsWlrCWX1L6JNT4iS4ir6kkhKYM8dZ/H7nJ+vPfw6jRh2ZIc/jARHWf+Xh8y899OjtIae7U1a5fv16uO02tKwM8Xphxgzo0wfi452kF1g+/TyeFR/FkTsgjh/nVl1HfLyTFIYODWpG/vBnbyrgXwkrAu9N05q8N/ffX2U7nYBbLjjy/uaE2sYppzizAx7jF0croF3Q3ympokp5uZNk9u+HRx91+jkr3t+LLnLmfDp8GE9REZ6iIko3HWbVO0Uk+Q6T4imic7si0uO/h6IdcPCw88Nk/37wOyffiL+c7G+WONcUBBJvRpMEvtydSIk/gXxPOp1PTySjeuL94gunDy2QeFN6d4XTT3eSaGDZn1fK5hVlxPnLSPSU0v1HRaQlHXLWl5Y6t3v2gAb6BtVHbvES8Dau8t6UJsBeQAFRaL4Z2BFUIT+fKv2LdaGqDWbp27ev1tfixYvrvY1wiaVYVGMrnrDE8uGHqg895NzWYxvf/OIX9d6GJierxsU5t3XdVjhiqYinvu9LOOOpr8D76/N4jvn+Hvdl1+DvVNNtRCKW424nsI0+4Nc6fudG/Us/koslGHfFUjwnXSxh+lKPpfdFNYbiiaXkG2OxtIFtWsfvXOsiM+ZEcNy+QVMv4eriDcffKcZi2Q55dX26jWpljDHGFZZgjDHGuMISjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyKaYEQkQ0TmiUihiGwRkdFHqSciMkVE9gWWKSJHhoATkV4i8qmIHA7c9gq1HWOMMdET6RbMdKAUyASuAp4SkZwQ9cYBlwI9gR7AcOBGABFJAN4CZgNNgReBtwLlxhhjYkTEEoyIpAAjgXtUtUBVlwNvA9eEqH4dME1Vt6nqdmAaMCawbjDOGGqPqWqJqv4ZZ0DWn7j8EowxxtRCJK/k7wyUq+rGoLLPgUEh6uYE1gXXywla94WqBg8D/UWg/N3qGxKRcTgtIoACEfm6buFXao4zRlwsiKVYILbisVhCi6VYILbisVhC61LXJ0YywaQCh6qVHQQaH6XuwWr1UgPHYaqvO9Z2UNVngKNMCVV7IrJaVfuFa3v1EUuxQGzFY7GEFkuxQGzFY7GEJiJ1nuMkksdgCoC0amVpQH4N6qYBBYFWS222Y4wxJkoimWA2AvEi0imorCewLkTddYF1oeqtA3oEn1WGcyJAqO0YY4yJkoglGFUtBN4EJotIioicDVwCvByi+kvAHSLSRkRaA3cCswLrlgA+4DYRSRSRWwLlH7gZf5CwdbeFQSzFArEVj8USWizFArEVj8USWp1jieiUySKSATwPDAX2Ab9R1f8VkYHAP1Q1NVBPgCnALwJPnQlMqjiwLyK9A2XdgA3AWFX9V8ReiDHGmOOKaIIxxhjTcNhQMcYYY1xhCcYYY4wrLMHUQOBkgucC46fli8gaEbkwBuLqJCLFIjI7BmK5UkQ2BMaZ+yZwXC0acXQQkfki8r2I5InIEyISkeu9ROQWEVktIiUiMqvauiEi8lVg/LzFItI+GrGIyJki8r6I7BeRPSLyuoi0ikYs1ercKyIqIue5Gcvx4hGRRiLypIjsFZGDIrI0irFcEfifyheR9SJyqcuxHPN7ri6fYUswNRMPbMUZdSAd+D3wmoh0iGJM4Izt9kmUY0BEhuKclPFfOBe8ngN8G6VwngR2A62AXjh/s5sjtO8dwIM4J7JUEpHmOGdQ3gNkAKuBudGIBWf8vmeADkB7nOvHXohSLACIyKnA5cBOl+OoSTzP4PyNTgvc/joasYhIG5zxFu/Auc7vLuB/RaSli7Ec9Xuurp/hSF7Jf8IKnGJ9f1DR30VkE9AX2ByNmETkSuAA8CHwo2jEEOQPwGRV/SjweHsUY+kIPKGqxUCeiLzLkWGGXKWqbwKISD+gbdCqy4B1qvp6YP39wF4R6aqqX0UyFlX9R3A9EXkC+KcbMRwvliDTgUk4Pw5cd7R4RKQrMAJoq6oVo458Go1YAvcPBP293hGRQuBUnB9QbsRyrO+5ZtThM2wtmDoQkUycsdWicnGniKQBk3F+3USViMQB/YAWIvIfEdkW6JZKjlJIjwFXBro62gAXEmKMugirMrZe4B/5GyKU+I7jHKJ4kbKIXA6UqOr8aMUQ5AxgC/CHQBfZlyIyMkqxrAY2iMgIEYkLdI+V4Iy7GBHVvufq9Bm2BFNLIuIF5gAvuvXrswYeAJ5T1W1R2n+wTMALjAIG4nRL9cZpXkfDUpwP/SFgG84/6v9FKZYKtRo/L1JEpAdwL073SzT23xh4CLg9GvsPoS3QHedv0xq4BXhRRE6LdCCq6sO54Px/cRLL/wI3Br7YXRfie65On2FLMLUgIh6ckQdKcT580YihF3Ae8Gg09h9CUeD2L6q6U1X3Av8D/DTSgQT+Pu/i9BWn4IxI2xTn+FA0xdz4eSLyI+AfwO2quixKYdwPvKyqm6O0/+qKgDLgQVUtVdV/AouBYZEOJHCyw59wpidJwDkuMlMiMLniUb7n6vQZtgRTQyIiwHM4v9hHqmpZlEIZjHOA9jsRyQMmACNF5LNoBKOq3+O0FIKv2I3W1bsZQDucYzAlqroP5wB2xJNdNVXG1hNnbqRTiV4Xa3tgIfCAqoYaqilShuAM+ZQX+Cxn4xxUnhSleEJ1P0Xrs9wLWKqqq1XVr6qfAKtwfly65hjfc3X6DFuCqbmncM4sGa6qRcer7KJncP6wvQLL08A7wPlRjOkF4FYRaSkiTXHOvPl7pIMItJ42ATeJSLyINMGZvC4i/daBfSYBcUCciCSJc4r0PKC7iIwMrL8XZ04j17pYjxZL4LjUBzhJ+Gm39l+TWHASTHeOfJZ34MxcOz1K8SwFvgPuDtQ5GzgXeC8KsXwCDKxosYgzPNZA3P8sH+17rm6fYVW15TgLzumcChTjNBUrlqtiILb7gdlRjsGLcwbQASAP+DOQFKVYeuEMiPo9zoRNrwGZEfxbaLXl/sC684CvcLphlgAdohELcF/gfvDnuCBa70u1epuB86L8d8oBVgKFwHrgZ1GM5RbgPzjdUN8Cd7ocyzG/5+ryGbaxyIwxxrjCusiMMca4whKMMcYYV1iCMcYY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY04Q4syXMiracRhTU5ZgjKkBEZkV+IKvvnx0/Gcb0zDZfDDG1NxC4JpqZaXRCMSYE4G1YIypuRJVzau27IfK7qtbROSdwJSyW0Tk6uAni8jpIrJQRIrEmbJ4loikV6tzXWAekhIR2SUiL1aLIUOcaY4LReTbEPu4N7DvksAgki+58k4YUwOWYIwJnz8Ab+OMh/YM8FJgpsKK0Wffwxnb6QzgZ8BZBE2VKyI3AjNwBg/tgTMK9Npq+7gXeAtnZNu5wPMi0i7w/JE4o2vfDHQCLgY+duF1GlMjNhaZMTUgIrOAq3EGAgw2XVUniYgCM1X1hqDnLATyVPVqEbkBmIozHW9+YP1gnPlGOqnqf0RkG87Apb85SgwK/FFV7w48jseZWG2cqs4WkTtwRiPurtGbTsKYSnYMxpiaWwqMq1Z2IOj+ymrrVgIXBe6fhjO8efAETR8CfqCbiBwC2gCLjhND5XDtqlouInuAloGi13Fmh9wkIu/hTL72tqqWHGebxrjCusiMqbnDqvqfasveMGy3Nt0I1VsmSuD/WFW3Al1wWjGHgGnAp4HuOWMizhKMMeFzZojHGwL3NwCnB+ahr3AWzv/gBlXdDWzHmYSrzlS1WFXfUdVfAz/Gmd/k7Pps05i6si4yY2ouUUSyqpX5VHVP4P5lIvIJzmRMo3CSRf/Aujk4JwG8JCL3Ak1xDui/qar/CdT5b+BREdmFM0tpI2CIqk6rSXAiMgbnf3oVzskE/w+nxfPvWr5OY8LCEowxNXcesLNa2XagbeD+/cBInBk99wD/pc5c6qjqYRE5H3gM58yuYpyzwW6v2JCqPiUipcCdwBRgPzC/FvEdACbhnEzgxZmR8TJV3VSLbRgTNnYWmTFhEDjD63JV/Wu0YzEmVtgxGGOMMa6wBGOMMcYV1kVmjDHGFdaCMcYY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY4wxrvj/LfV5tIs8PbYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "HAS8HZnX5jk9", "outputId": "f53af0f5-72cc-43d4-d48c-d6214f166d76", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1hUV/rA8e87VAEVRMCCgopgb1iwY6KmmmRTTKJpm2I2ZZPdbNpudhM3m7ZJNpst6b+42SQmWZNo6sYWIZbYQEVFsCGoiCBFkA4z5/fHDOxAAClT4XyeZx6dc+89c+Yy8M4959zzilIKTdM0TbM3g7MboGmapnUNOuBomqZpDqEDjqZpmuYQOuBomqZpDqEDjqZpmuYQOuBomqZpDqEDjqZpmuYQLhlwROR+EUkSkSoRee88+/5aRE6LSImILBMRH6ttkSKSICLlIpIuInPt3nhN0zStSS4ZcIBTwDPAspZ2EpGLgMeBC4EIYDDwR6tdPgZ2A8HAE8BnIhJijwZrmqZpLRNXXmlARJ4BwpVStzWz/SMgUyn1O8vzC4HlSqk+IhIN7AN6K6XOWbZvsmx/0yFvQNM0Tavn6ewGdNBI4Eur5ylAmIgEW7Zl1AUbq+0jm6pIRJYASwB8fX1jBw4caJ8WuxGTyYTB4KoXwY6jz4M+B3X0eWj5HBw6dChfKdVsL5K7B5wAoNjqed3/uzexrW57/6YqUkq9DbwNEBMTow4ePGjblrqhxMRE4uPjnd0Mp9PnQZ+DOvo8tHwORCSrpWPdPVSXAj2sntf9/1wT2+q2n0PTNE1zOHcPOKnAWKvnY4FcpVSBZdtgEeneaHuqA9unaZqmWbhkwBERTxHxBTwADxHxFZGmuv/eB+4QkREiEgj8HngPQCl1CNgDPGU5/mfAGOBzh7wJTdM0rQGXDDiYA0cF5inPN1n+/3sRGSgipSIyEEAptRp4EUgAjgNZwFNW9dwATASKgBeAa5VSZxz2LjRN07R6LjlpQCm1FFjazOaARvu+ArzSTD2ZQLztWqZpmqa1l0sGHFdXUlJCXl4eNTU1zm6KXfXs2ZO0tDRnN8Nh/P39CQ8P7/LTXjXNXnTAaaOSkhJyc3Pp378/3bp1Q0Sc3SS7OXfuHN27dz//jp2AyWQiOzub/Px8QkNDnd0cTeuU9Fe5NsrLy6N///74+fl16mDT1RgMBsLCwigubnzrlqZptqIDThvV1NTQrVs3ZzdDswMvLy9qa2ud3QxN67R0wGkHfWXTOemfq6bZlw44mqZpmkPogKNpmqY5hA44mqZpmkPogNMFiEiLj9tuu81pbYuMjOTll1922utrmuY4OuA4yfLlEBkJBoP53+XL7fdaOTk59Y933nnnJ2V/+9vf2lRfdXW1PZqpaVonpwOOEyxfDkuWQFYWKGX+d8kS+wWdPn361D8CAwMblJWVlXHLLbfQp08f/P39mTBhAt98802D4yMjI1m6dCm33347gYGBLF68GIBly5YxcOBA/Pz8WLBgAa+//vpPZnp9/fXXxMbG4uvry6BBg3jiiSfqA1Z8fDxZWVk88sgj9VdbmqZ1XjrgOMETT0B5ecOy8nJzuaOVlpZyySWXsG7dOlJSUrjmmmu4+uqrSU9Pb7DfK6+8wrBhw0hKSuK5555j69at3Hnnndx3333s2bOHK664gqeeeqrBMWvWrGHx4sXcf//9pKamsmzZMj777DN+97vfAbBy5UrCw8N58skn66+2NE3rxJRS+tHoER0drZpz4MCBZre1lohS5mubhg+RDld9Xp9++qky/9ibN2XKFPWnP/1JlZSUKKWUioiIUJdffnmDfW644QZ10UUXNSi76667GtQ9c+ZM9fTTTzfYZ9WqVcrf31+ZTKb6ul966aV2vx9ba+rnm5CQ4PiGuBh9Dsz0eWj5HABJqoW/rfoKxwkGDmxbuT2VlZXx6KOPMmLECIKCgggICCApKYnjx4832G/ixIkNnqenpzN58uQGZVOmTGnwPDk5mWeffZaAgID6x6JFiygrK+P06dP2eUOaprksvXinEzz7rHnMxrpbzc/PXO5oDz/8MKtXr+bll19m6NCh+Pn5ccstt/xkYoC/v3+b6zaZTDz11FNcd911P9kWEhLS7jZrmuaedMBxAsuYO088AcePm69snn32f+WOtHnzZm655RauueYaACorKzl69CjR0dEtHjds2DB27tzZoGzHjh0Nnk+YMIH09HSioqKarcfb2xuj0djO1mua5k50wHGSxYudE2Aai46OZtWqVVx55ZV4eXnxxz/+kcrKyvMe98ADDzBjxgxeeuklrrrqKjZu3MiqVasa7PPkk09y+eWXExERwcKFC/H09GT//v3s2LGDF198ETDPgNu0aRM33XQTPj4+9O7d2y7vU9M053PZMRwR6SUiq0SkTESyRGRRM/t9Z0k7XfeoFpF9VtszRaTCavtax70L1/fKK68QGhrKzJkzueSSS4iLi2PmzJnnPW7q1Km88847/P3vf2fMmDF88cUXPPbYY/j6+tbvc9FFF/Htt9+SkJDA5MmTmTx5Mi+88AIDrQarnn76aU6cOMGQIUN0N5umdXKufIXzGlANhAHjgG9FJEUplWq9k1LqEuvnIpIIbGhU1wKl1Ho7ttVtXHvttZgnk5hFRESwfn3DU/Pwww8D5gRsAJmZmU3Wdfvtt3P77bfXP//1r3/9k+6z+fPnM3/+/GbbExcXR0pKSpveg6Zp7sklA46I+APXAKOUUqXAZhH5CrgZeLyF4yKBmcBt9m+l9tJLLzFv3jwCAgJYv349b775Js8995yzm6VpmotyyYADRAO1SqlDVmUpwOzzHHcLsEkpldmofLmIGIDdwCNKKf2V2gaSkpJ4+eWXKS4uZtCgQTz//PM8+OCDzm6WpmkuSqy7V1yFiMwEPlVK9bEquwtYrJSKb+G4I8AzSqn3rMqmA7sAAR60PIYppc42OnYJsAQgJCQkdsWKFU2+Rs+ePVucddWZGI1GPDw8nN0Mhzpy5MhP0kyXlpYSEBDQoOxwUS1pBSZGBHsQFdT5z1FT56Ar0ueh5XMwZ86cZKXUxCY34rpXOKVAj0ZlPYBzzR0gIjOAPsBn1uVKqS1WT58XkVsxd7t93Wi/t4G3AWJiYlR8fHyTr5OWlkb37t1b9Sbc3blz57rMe63j6+vL+PHjG5QlJiZi/XlIziri+TU/YlLw3ywjy++MIzYiyMEtdazG56Cr0uehY+fAVWepHQI8RWSoVdlYILWZ/QFuBVZaxnxaojBf7Whau6zadRKTpWOgqsbEtowC5zZI09yESwYcpVQZsBJ4WkT8Ld1iVwIfNLW/iHQDFgLvNSofKCLTRcRbRHxF5BGgN7CliWo0rVVSTv6vN1YBw/t2ratATWsvlww4FvcC3YA84GPgHqVUqojMFJHGVzFXAWeBhEbl3YE3gCIgG7gYuEQppb+Sau2y5Ug++7JL+Pn0SG6fMQgPgbWpuc5ulqa5BVcdw0EpVYg5kDQu3wQENCr7GHNQarxvKjDGXm3UuhalFC+tOUjfnr48dvEwfL08MADvbjnGTXERjOrf09lN1DSX5spXOJrmUtan5bHnxFkevHAovl7mmWkPzB1KLz9vln6ViivO+NQ0V6IDjiO9+CIkNO71ayQhwbyfm/rss88aZO587733OjyNNDExEREhPz+/o81rN5NJ8fKagwzq7c81seH15T18vXjkohiSsor4KuWU09qnae5ABxxHmjQJFi5sPugkJJi3T5pk85e+7bbb6tM4e3l5MXjwYB5++GHKysps/lrWrr/+ejIyMlq9f2RkJC+//HKDsmnTppGTk0NwcLCtm9dqX+89xcHcc/x6XjReHg1/ba6bOIBR/XvwwnfplFfXOqmFmub6dMBxpDlzYMWKpoNOXbBZscK8nx3MnTuXnJwcMjIyeOaZZ3j99dfr102zVltba7PuoW7duhEaGtqhOry9venTp0+DKydHqjUpXll3iGF9unP56L4/2e5hEJYuGElOcSVvJh51Qgs1zT3ogONoTQUdBwQbAB8fH/r06cOAAQNYtGgRixcv5osvvmDp0qWMGjWK9957jyFDhuDj40NZWRnFxcUsWbKE0NBQunfvzuzZs0lKSmpQ5/vvv09ERAR+fn5cfvnl5OY2nLHVVJfaf//7X6ZMmUK3bt0IDg5mwYIFVFZWEh8fT1ZWFo888kj91Rg03aW2cuVKRo8ejY+PDwMGDODZZ59tECQjIyN55plnuPvuu+nRowfh4eG89NJL7Tpvm7NrySoo55GLYjAYmg56EyN7ccXYfry1MYMTheVN7qNpXZ0OOM5gHXSefNIhwaYp3bp1o6amBoBjx47x0Ucf8emnn5KSkoKPjw/XXXcd2dnZfPPNN+zevZtZs2ZxwQUXkJOTA8D27du57bbbWLJkCXv27GHBggU8+eSTLb7m6tWrueKKK5g3bx7JyckkJCQwe/ZsTCYTK1euJDw8nCeffJKcnJz612ksOTmZ6667jquvvpp9+/bxwgsv8Pzzz/PPf/6zwX5//etfGT16NLt27eKxxx7j0UcfZevWrW06R5U1Rr48UsOEgYFcMKzlK7XfXjoMgwiPfpbCawlHSM4qatNraVqnp5TSj0aP6Oho1ZwDBw40u63N/vAHpcD8r53deuut6rLLLqt/vn37dhUcHKwWLlyonnrqKeXp6alOnz5dv/37779X/v7+qry8vEE9Y8eOVX/+85+VUkrdeOONau7cuQ2233HHHcr8sTL717/+pfz9/eufT5s2TV1//fXNtjMiIkK99NJLDcoSEhIUoM6cOaOUUmrRokVqzpw5DfZ56qmnVP/+/RvUc8MNNzTYJyoqSv3pT39q9rWV+unP96kv96mIx75R72051uJxdR7/LEVFPPaNinzsGxXz+/+qpMzCVh3n6hISEpzdBJegz0PL5wBIUi38bdVXOM6SkABvvAF/+IP53/PNXrOB1atXExAQgK+vL1OnTmXWrFn84x//ACA8PJywsLD6fZOTkykvLyckJISAgID6x/79+zl61DxOkZaWxtSpUxu8RuPnje3evZsLL7ywQ+8jLS2N6dOnNyibMWMG2dnZlJSU1JeNGdPwFqx+/fqRl5fX6tfZfCSf937MAuD579JadcXSp6c5AZ2ibtkb582s0zRX47I3fnZqjcds5sxxSLfarFmzePvtt/Hy8qJfv354eXnVb/P392+wr8lkIjQ0lM2bN/+knh49Gq+r6jqsJxZYv7+6bSaTqdV1fbz9eP3/a2rNa6adb5HOGUNDeCPxKJW1JhTmhT6ra014e+rvdpqmfwscrakJAi3NXrMhPz8/oqKiiIiI+Mkf48YmTJhAXl4eBoOBqKioBo+6WWfDhw9n27ZtDY5r/Lyx8ePH8/333ze73dvbG6PR2GIdw4cPZ8uWhsvhbd68mfDwcJuubl1cYR7fMgBengbiBp9/WnZsRBDL74rjkYuiuW5iOBvSz3DLsu2cLa+2Wbs0zRaSMwv5x4bDDh1r1Fc4jtTSbDTroOOECQSNzZ07l7i4OK688kpefPFFhg0bxunTp1m9ejVz585l5syZPPDAA0ybNo3nn3+ea6+9lsTERFatWtVivU888QQLFiwgKiqKRYsWoZRi7dq13H333fj5+REZGcmmTZu46aab8PHxoXfv3j+p4ze/+Q2TJk1i6dKlLFq0iJ07d/KXv/zFptlGa40m9mUXEx8dQm91lhvnTmp1CoLYiKD6facP6c2jn+3lZ6//yLu3TmRwSNfOpaI5XnJWEdsyCogbHEy/QF+2HCng65Rsfjhk7u59VQ7z1xvGcsXY/nZvi77CcaSdO1sOJnVBZ+dOx7arCSLCZ599xgUXXMBdd91FTEwMCxcu5ODBg/Tr1w+AuLg43n33Xd544w3GjBnDypUrWbp0aYv1XnrppaxatYrvvvuO8ePHM3v2bBISEjAYzB/Fp59+mhMnTjBkyBBCQkKarGPChAl8+umnfP7554waNYrHH3+cxx9/nPvvv99m73/3ibMUV9Rw3cQBXD7Eu935bq4a35+P7ppCcUUNC/6xmcc+36tnr2kOk5xVxI1vb+OlNQe59o0fmfr8Bh7+NIUdx/73GTQqxQMf7+HuD5LYfdy+n02XzPjpbDExMergwYNNbktLS2P48OEObpFzdMUEbHU/3z+vTuedjRnsenIeu7Zt6XDSrf/uzeG+j3ahAF8vg9slbdOJx8zc7Ty8vOYg/0w4Uv98TkwIj148jLKqWm56dzs1tSY8PQwsGNuXdQfyKK6oYcqgXswdEUZ1rZG4wb1/8jlt6RyIiFtm/NQ0p0pIz2NiZBA9fFse62qtYwX/W0KoupUTEDSto+rm0BgEvD0N3H/BUIb3NU/6WX5nXH1XW2xEEKVVtXyy4zivJxxl+7FCAHy9jtj0y5EOOJrWSPbZCtJPn+N3lw6zWZ1xg4Px9jRQVWvCINKqCQia1lFH8koJ8vPijhmDmDqk4dWK9VgjQICPJ3fOHEx5tZG/rjuEovWzM1tLj+FoWiMJ6eZ7dc63skBbxEYE8dGdU/Dz9mB6VLC+utHsrrLGyA+HznDJ6L7cf8HQVn/mpkf1xsfLgIe0fnZma+krHE1rJCE9jwG9ujHExjPKYiN7MW1IMBn59l2hW9MAtmYUUF5tZN6IsPPvbCU2Iugn3W22ogNOOyilnLZysWY/5uU3YMvRfK6fOMAuP+PxA4NYn5ZHcXkNPf1sMz6kaU1ZfyAXP28PprbjCqVxd5utuGyXmoj0EpFVIlImIlkisqiZ/ZaKSI2IlFo9BlttHyciySJSbvl3XEfa5eXlRUVFRUeq0FxUTU0NlUZFZY2JeBt2p1kbPyAQgD0nz9qlfk0D85en9Wm5zBoaUp+d1hW4bMABXgOqgTBgMfCGiIxsZt//KKUCrB4ZACLiDXwJfAgEAf8GvrSUt0toaCjZ2dmUl5frlMKdiMlkIjc3l315Nfh6Gdr1rbA1Rof3RAS73++gdW37sovJLalqc3faT9g4S7FLdqmJiD9wDTBKKVUKbBaRr4CbgcfbUFU85vf4qmUl07+LyMPABcDq9rStbh2xU6dO1S/t31lVVlbi6+vr7GY4jL+/P//aVcD0Ib3t9q2wu68X0aHd2X1cX+Fo9rP+QC4GgTkdvVKvy1Lc3A3r1quntIJLBhwgGqhVSh2yKksBZjez/wIRKQRygH8qpd6wlI8E9qqGlyJ7LeUNAo6ILAGWAISEhJCYmNjhN+HuSktLf5I8rTM7VWois6CCWWHGBj//0tJSm34ewryqSDp2joSEBLcZC7T1OXBX7nIeVu2sICrQwN6dP3asIhECf/c7RvzsZxx46inOjh9ffw4Cd+9mxB//aC4XgVacF1cNOAFASaOyYqCp295XAG8DucAU4HMROauU+thST3Fr6lFKvW2ph5iYGOVOdxPbi7vdVd1Rb288CqRz9xUz6B/Yrb7c1ufhtN9xNq7cR8SoSW6ztlpX+yw0xx3Ow8mick6sTuB3lw4jftaQjlcYHw/jxjHOciWTGBBAvFLw3HOwahXj2rDuo6uO4ZQCjdfA7wGca7yjUuqAUuqUUsqolPoR+BtwbVvr0bQN6XkM69O9QbCxh3EDLRMHTuhuNc321h8wp3mfN6KP7Sq1Wlw4ctmydi8y7KoB5xDgKSJDrcrGAqmtOFYBdf0UqcAYadhvMaaV9WhdSEllDUmZRR3v826FoaHd8ff20OM4ml2sT8tjSIg/g3r7n3/ntpgzB+65h8gPPoB77mnXivYuGXCUUmXASuBpEfEXkenAlcAHjfcVkStFJEjMJgMPYJ6ZBpAIGIEHRMRHROqWE95g9zehuZVNh/KpNSmbri7QHA+DMHZAILtP6Jlqmm2VVNawLaOAuR2dndYUS5bizJtvbneWYpcMOBb3At2APOBj4B6lVKqIzBSRUqv9bgCOYO4mex/4s1Lq3wBKqWrgKuAW4CxwO3CVpVzT6m1Iz6NnN6/6+2TsbdyAQNJzzlFR3XKyOU1rix8OnqHWpJg33MYBx2o2Wubtt7c7YaSrThpAKVWIOVg0Lt+EeTJA3fMbz1PPbiDW5g3UOg2TSfHDoTxmR4fg6eGY72DjBwZRa1LsP1XMpMheDnlNrfNbdyCXYH9vxg+04SoBjRNHJia2O2GkK1/haJpD7MsuJr+0mjnDmk74Zg/j6lYc0OM4mo3UGE0kHMzjgmGheBhsNN2+tVmKW3mlowOO1uUt354FQKBfuxegaLOQ7j6EB3XT4ziazew8Vsi5ylrbjt9YZSlevhwiI+GCC2YTGQnLl9PmLMUu26WmaY6QnFXEp0knAbjnw2SHZuIcPzCI5MxCh7yW1vmtS8vFx9PAzKG9bVfpo48C5uCyZAmUlwMIWVnm5wCLF8/RXWqa1hpf7cmmbhmKumRTjjJuQCCniis5XVzpsNfUOielFOsO5DIjqjd+3ra/jnjiibpg8z/l5ebyttABR+vSzpRWAdgl2dT5jK+/AVR3q2kds2p3NieLKhgaZp+VK44fb1t5c3TA0bqsyhojW44UMD2qNw/Nj3FodxrAiL498PIQdusVB7QOSM4q4tHP9gLwry2ZJGfZ/gvMwIFtK2+ODjhal7XuQC7FFTX8YvZg7psT5fC0z75eHozo11OvOKB1yLaMAmpN5o7hWqN9uoWffRb8/BqW+fmZy9tCBxyty1qRdIL+gd2YNsSGg6xtNH5AIPtOFlNrNDmtDZp7mzzIfB+XYL9u4cWL4e23ISICRBQREebnixe3rR4dcLQuKftsBZuP5HNNbLjt7lloh/EDA6moMXIwV68nq7WPn7c5d9Olo/vatVt48WLIzIQNG34gM7PtwQZ0wNG6qM+TT6IUXBcb7tR2jB9g/uOgu9W09qobs3n8kmEO7xZuKx1wtC7HZFKsSDrB9KhgBvTyO/8BdjSgVzd6+XvrVAVauyVlFhHWw3wjsavTAUfrcrZlFHCyqIKFEwc4uymICOMHBLL7uJ4arbVPclYREyN6uUX2WB1wtC5nRdIJuvt6ctFIGyao6oBxAwI5eqaM4ooaZzdFczM5xRVkn61w+a60OjrgaF1KcUUN3+0/zZXj+uHr5eHs5gDUr+z7zDcH7HIPhdZ5JWWaPy8TI3XA0TSX81XKKapqTVw/sY13rNmRsiyu81nySRb/3zYddLRWS84qopuXB8P79nB2U1pFBxytS/k06QTD+nRnVH/X+QXde7IYMOdGd/R6bpp7S8oqZNyAQLwclMepo9yjlZpmA2k5Jew9WczCiQNcaoA1bnAwnpZ7gTw9HLuem+a+SqtqOXCqhElu0p0GLhxwRKSXiKwSkTIRyRKRRc3s94iI7BeRcyJyTEQeabQ9U0QqRKTU8ljrmHeguZoVSSfw8hCuGt/f2U1pIDYiiFcWjgXg5qkRbjMArDnXnuNnMSmIdaOMsS4bcIDXgGogDFgMvCEiI5vYT4BbgCDgYuB+Ebmh0T4LlFIBlsd8ezZas53krCJeSzhikzGNbRn5fLzjOJMietHL33GJ1lrrinH9GRziT3qOXnFAa52krEJE/rfquDtwyYAjIv7ANcAflFKlSqnNwFfAzY33VUq9qJTapZSqVUodBL4Epju2xZqtbc8o4Pq3tvKXtQc7PJD+3b4cFr2zncoaE0lZhS47KD9/RB+2ZRRQXK6nR2vnl5xVRExYd3r4ejm7Ka0mSqnz7+VgIjIe2KKU8rMqexiYrZRa0MJxAuwC3lJKvWkpywS6YQ6uu4FHlFIpTRy7BFgCEBISErtixQrbvSE3VVpaSkCAbfJrHCkykl5oZFgvD6KCWp6OXGtS/GFLBTll5s+mAbh6qBeXD2nblUlpteLro9Wsy6qlbmnM9tRly/PQkiNnjTyzrZIlY3yY1s+1kvE66hy4Olc5DyaluHd9OdP6eXLLSB+HvnZL52DOnDnJSqmJzR3rWp/q/wkAShqVFQPdz3PcUsx/U/5lVbYYcxAS4EFgjYgMU0o1WEtEKfU28DZATEyMio+Pb2/bO43ExERscR52HCvghbXbMSmFt6exxQUGa4wm7lu+i5yycjwMgtGkEBFunDupVWMbyVlFbD58hrxzVXyVcoqyqlrih4Ww5UgBtUYTXp6GVtdVx1bn4XxmmRRvpX7PSRVEfHys3V+vLRx1Dlydq5yH1FPFVK7ZzBXTRhHv4DHJjpwDVw04pUDjeas9gGY7uEXkfsxjOTOVUlV15UqpLVa7PS8itwIzga9t11ytJe9uPlafr6O61sS2jPwm/+DXGE388qPdrD2Qy9IFIxgdHshLa9LZllHIqVbcTZ2cVcQNb2+lxmh+rYkRQTx39Wiiw7qTnFXEtowC4gYHu+ygvMEgzBsRxhe7s6msMbrMjama66nrFnbVz3JzXHIMBzgEeIrIUKuysUBqUzuLyO3A48CFSqmT56lbYb7a0RzAaFLsPVlcf8JNCrYcKeBseXWD/WqMJh74eDerU0/z5OUjuG36IGIjgvjgjilMGBjI71bu43hB+U9foK5ek+LPq9Pqg41BYM6wUKLDzBfFsRFBTkmy1lbzR4RRXm3kx6P5zm6K5sLcacFOay4ZcJRSZcBK4GkR8ReR6cCVwAeN9xWRxcBzwDylVEajbQNFZLqIeIuIr2XKdG9gS+N6NPtYd+A0OcWV/HpeNA/Pj2bRlIHsOFbIRa9u5IdDZwBzlsJffbKH7/af5veXDef2GYPqj/fyMPC3G8aDwC8/2U1NE4nKqmtNPLRiDzuOFeFhEDwEvO2UiMrepg4JJsDHk7Wpuc5uiubC3GnBTmuu2qUGcC+wDMgDCoB7lFKpIjIT+E4pVTdq9QwQDOy0OvkfKqV+gXnM5w1gCFAJ7AEuUUrpW7kdQCnFGz9kMLCXH/fNiapPdLZo8kB+/Z893LpsB5eM6sOx/DLST5/j95cN586Zg39Sz4Befvz5mjHcu3wXL689yG8vGV6/7VxlDb/4MJktRwp45KIY4gb1YtuxQpfuOmuJj6cH8TEhrE/LxWhSTk0Op7mmU2fNC3beOXPQ+Xd2MS4bcJRShcBVTZRvwjypoO55s2ddKZUKjLFLA7Xz2n6skJQTZ/nTVaMa/OEc1b8nX/9yBg9/msI3e1TAgg0AACAASURBVHMA8DRI/SKWTbl0dF9unDyQt37IYPqQ3syKDiG3pJJbl+3gSF4pf7luLNdYkqm5041wTZk3Ioxv9uaw50QRsRHu/V4020uyjN9MdMPPhkt2qWmdw5s/HKV3gHeTWTV9LQsO1sUhpdR51xB78vIRRIcF8MuPd/HY5ylc9vdNnCgsZ9ltk+qDTWcwZ1goXh6iu9W0JiVnFuLn7cHwvuebtOt6dMDR7CItp4TEg2e4bVpks7Ot4gYH4+1pwEPAqxVjLt28Pbg3Poriilr+s/Mk+aXVLL1iJLOiQ+zxFpymh68XcYODWZN6Gle8T05zrqSsIsYNCMTTTRbstOZ+LdbcwtsbM/Dz9uCmuIhm94mNCGL5nXE8ND+mxXtzrGWfraif8WYQyDtX1eL+7mr+yD5kFpRzJK/U2U3RXEhpVS1pOSVMdMPxSdABR7ODk0XlfJVyihsnDyTQr+U7+ts6XTlucDA+Xga3nonWGvOGhwGw9oDuVtP+xx0X7LTmspMGNPf17uZjCHDHDNvPoqm7KnL1mzg7qk9PX8YOCGRt6mnumxPl7OZoLsIdF+y0ZvMrHBH5u4h800R5DxFZKiLDrcp+JSL7RERfaXUSRWXVfLLjBFeM60e/QPvclOYuN3F21PwRYaScLOZ0caWzm6K5iIT0PHr7e3M41z27Wm36h15EhgC/wLymWWMTgacA66VN3wJCgFtt2Q7NeT7YlkVFjZG7Zw1xdlPc3kUjzd1q69J0t5oGOzMLSTlZzJnSardNRW7rK4tfASlKqaQmto0HqoADdQVKqQrgfeBhG7dDc4KtR/N5I/EosRFBxPRxvymbrmZISAD9evry7qYMt/zjotnWF7uz6//vrqnIWxVwRCRKRGpE5OlG5W9YMm1OFBEf4CbgoyaOTwNeBnyAGhFRIvK5ZfMnwAgRmdahd6I5VXJWETe/u4OKGiP7ThbrP5A2sOv4WXLPVZFZUM6id9zzG61mO+XVRoBW30bgiloVcJRSR4D/A34lIsEAIvIkcDvwM8sVTRwQCGxqoopbgAzMKzRPtTwesmzbg3kV6Ivb/zY0Z9uWUVC/IrTR5J7fvlzNtoyC+vtwqmpNrE097eQWac60L7uY0f17tOk2AlfTli61pwEP4HERuRPzeMzNSqn1lu1xmFdi3tvEsSlAOLBBKbXN8sgCUEqZLNvj2vkeNBdQd9ez4L7fvlxN3Y2xdasxfJ1yioLSznnfkdayY/llHMkr5eoJ4W49YabV06KVUjki8irwG8txDyilrNNi9gNKlFLVTRw+EvDGnAitKWeA6Na2RXM95yprAbgpLoKrxvd3218IV2I9BbxnNy/+9M0Bfv7eTj66K44AH31HQ1ey3nI/1lzL/Vnuqq2f2sOYx2E2K6Vea7TNF/OkgKZMwHz1s6eZ7RWY00BrbmrT4XwC/bxYesVIvcKxDcVGBNUH736Bvtz1fjJL3k/iXz+fhI+nTtDWVaxLy2VYn+4M6OXn7KZ0SKu71ETkQszTmLcC00Wk8SrMBZjHcJoyHjiqlGqcNrpOL0BnnHJTSik2H85n+pDeOtjY0QXDwnjp2jH8eLSAX32yB6NJr7PWFRSWVZOUWcj8Ee59dQOtn6U2AViFeeJAPHAceL7RbumAt4g0tWzvCKymQzdhEHCwNW3RXM+RvFJOl1Qyc2hvZzel07t6Qji/v2w43+0/zS8+TOa1hMN69lontyE9D5OCeSP6OLspHXbegCMiUcB3wFrgl5Yxmj8Cl4rILKtdN1r+ndxENWeBsSJykYjE1c10s9QfiHn8ZmMTx2luYNNh88XpDB1wHOLOmYP52fj+rDuQy0trDnH9W1v5eMdxqmvN2VCTs4p4LeGIDkSdxPoDufTp4cuo/j2c3ZQOa3EMR0T6YA40acBiy4wyMN+s+SjwAjANQCmVKSI7gAWY00NbexJ4F/gC81jPTGCzZdtlQDXmKyjNDW06fIZBvf0JD3Lv/mV3EhXqj2AeGK01KX67ch9//DqVISH+HDxditGk8PIw8PvLh1vyDgkGgcO559h/qoTxA4IYHd4TLw/BwyB4eRg4kFNCanYxU4f0bjDpIzmriG+OVtN9UJGeDOJglTVGNh4+w9UT+rtdOummtBhwlFKngZ/k/FVKGYHhPz2CN4C/ich9Sqlyq/33A1OaeZmbgE8bp30WkV6Yg9R8zOM7v1VKNXVTqWAOfHdaiv4PeFxZbmAQkXGWeoZjDpx3KKWam7ygtVFVrZFtGYVcN7HzJEBzB3GDe+PjdYSaWhOeHgYevDCK/NIa/rvvVP39UNVGE09+mdrk8e9vzWqh9kNEhfgzOCQAg0h9uutvMre57f0f7urHo/mUVxs7RXca2H616A+Bx4B7Ma8s0CJLMLgA87Tpxl7DfOUTBowDvhWRFEvaaGtLMKeiHov5C9864Bjwpoh4A18CrwKvA3cDX4rI0Gamb2tttCvrLBU1RmZE6e40R2pu1ezLxvRl8TvbqDaa8DQY+N1lw4gK6Y5RKb7ak83KXdkozLmELh3dlzkxoRhNivVpuaw7kEvdNIRakyKroJyswrL6AFa3nIoOOI6z7kAeAT6exA12z3QEjdk04CilakXk55inQbdGH+A2y0oG9UTEH7gGGKWUKgU2i8hXwM3A443quBX4i1LqpOXYvwB3AW9inuDgCbxqueL5u4g8jDnIrW7HW9Qa2XzkDB4GYeoQfaOno1lPmbYuW35X0+kbAnw8+XZfDjW1Jrw8Dfx8+qD67UNCA9h4+Ez9tr8sHEdsRBDJWUXc+M42qmtNGET0Db0OZLJ8EZgdHdJ5psArpVzugXkadXmjsoeBr5vYtxiYYvV8InDO8v9fA9812v8b4DdN1LMESAKS/Pz8FOarJf04z6PPza+osMV/dno79KN1D+9+w1SPuOuUd79hrd7m3W+Y6n/fB6rv7a85vf1d6eHdN1pFPPaN8h8R7/S2tOGR1NLfdlfNQxMANL5npxhoagniAMs26/0CLGM7jbc1W49S6m2l1ESl1MTw8HCnB11XeCQkJLS4vbC0Ct/+0Tx+21VOb6szz4M7Paqy0yjeuoKq7LRWb6vKTuOm2DC8QyI4dLrE6e+hq3wWnvnXl3gYhJM71zj9fbf2HJyPqwacUqDxHMAemBf5PN++PYBSZX73balHa6MfjxagFMwcGuLspmh2Nq2fJ54G4bPkk85uSpex7kAukyN7nTdNuztx1YBzCPAUkaFWZWOBpqbcpFq2NbVfKjBGGs4nHNNMPVobbTp8hu6+nowN7+nspmh21sNHmDMslJW7s6k1ms5/gNYhWQVlHMotZW4nWF3AmksGHKVUGeZ7eZ4WEX8RmQ5cCXzQxO7vAw+JSH8R6Yd5cdH3LNsSASPwgIj4iMj9lvIN9mx/V6CUYtPhfKYNCcbTwyU/RpqNXRcbzplzVfxw6Iyzm9LprbMs1tkZlrOx5sp/Ke7FvKBnHvAxcI9SKlVEZoqIdULvtzDn2dkH7Ae+tZShzFOfr8Kcj+cs5vw9Vyk9JbrDjuWXkX22ghm6O63LmDMslGB/bz5N0t1q9ra+kyzW2ZjLrnGulCrEHCwal2/CPBmg7rnCvOrBo83UsxuItVMzu6zNR8zL2czU9990GV4eBq4a35/3t2ZSWFZNL//OM7bgSs6WV7Mzs4h7Zg9xdlNszpWvcDQXtvFQPgN6dSMiuHN9A9Nadt3EcGqMii92Zzu7KZ3Wss3HMJoUA3p1vowtOuBobVZjNN9xPiMqpFOs76S13rA+PRjdv6eerWYnyVlF/DPBfB/8U1+ldroFWHXA0dos5cRZSqtqmaVXh+6SrpsYbl7o81TjW9y0jtp8+Ax1aY7qlhLqTHTA0dps4+F8DALThuiA0xVdMbYf3h4GPXnADny9zEvYGAS8PA2dbikhHXC0Ntt8+AyjwwPp6efl7KZoThDo5828EWF8uSe7PgePZhsZZ8rw8/LgV3OjO+XK3DrgaG2y8dAZdh8/y9BQf2c3RXOiayeGU1Rew/dpuc5uSqdRt2r33BFhPHDh0E4XbEAHHK0NkrOKuOPfO1HAV3tyOt2AptZ6s4aGENbDh0872eSBlrKl1iWis9fnftfxIgrKqpk/snPd7GnNZe/D0VzPtowCao3mEU2jSedG6co8DMLVE8J5M/EoL65O58LhYW7/WUjOKmLxO9uoqjXh6SE8eOFQBvU23/KXcaaUv284TK3Rfono1uw/jbeHgfiYUJvW60p0wNFaLW5wMHV5jTvjgKbWNiP6dkcBbyQeZdmWY24/5rAtI59Ky5hUjVHx8tpDTe5nj0R0SinWHshlelQwAT6d989y531nms2N6NsDAzBxUC8evXiYW/9x0TrueGEFYE6C0hmygZZU1ALm71TengZeuHoMI/r1QAQOnCrhkc9SqDEqxA6J6A7mnuN4YTn3xHe+1QWs6YCjtdqeE2cxKrh79mC3/sOi2Ubc4GA8DILRpPDycO8r3qyCMj7YlsXo/j24aGQfpg7p3eAzHh1mXtfsoQ+3cfycCXOYtZ21qbmIwIXDO293GuhJA1obJGcVAjBhoA42mjmd9R8uHw7Ag3Pdd1aV0aR4+NMUPER46+aJ3H9B0+8lNiKIx6f4Et6rGw9+sodzlTU2a8Oa1NNMGBhEaHdfm9XpinTA0VptZ2YR0WEBnSohlNYxiyZH4O/twcmiCmc3pd3e3ZzBzswill4xkn6BLa9f1s1TePX68eQUV/LUl7ZJq3WyqJzUUyVc1Ilnp9XRAUdrFaNJsSuriImRvZzdFM2FeHsamBbVm8SDZ1qVYtjVHDx9jpfXHGL+iDCuntC/VcfERgTxywuiWLk7my/3dHwR07rcN/NG9OlwXa5OBxytVQ6ePse5qlomRbpnt4lmP/ExIWSfreDomdLz7+xCqmtNPLRiD919PXnu6tFtWoj2/jlRTBgYyO+/2M/JovIOtWNtai7RYQEM6t35b6bWAUdrlSTL+M3ECH2FozVUd99I4kH3ygT6zw2HST1VwrM/G03vAJ82HevpYeDV68ejFDz0nxSMpvZd3RWVVbMjs5D5XeDqBnTA0VopKbOIPj18CQ/qfDk6tI7pH9iNoaEBbpV6+j87j/OPhCPMju7NxaPa98d+YLAfT185kh2Zhdz2rx3tWoHg+/Q8jCbVqVcXsOZyAUdEeonIKhEpE5EsEVnUwr6PiMh+ETknIsdE5JFG2zNFpEJESi2PtfZ/B51TUmYhsZFBOv+N1qTZ0SFszyikvLrW2U05rx3HCnj8830oBdsyCju0VE1ELz8MApsO53P9W1vZ3sZ0AmtTT9O3py+j+/dsdxvcicsFHOA1oBoIAxYDb4jIyGb2FeAWIAi4GLhfRG5otM8CpVSA5THfXo3uzLLPVnCquJJJbjrtVbO/+JhQqo0mth51/fwtn+86WX8XTa2xYzlnth0rrP9/rUlx30e72HvybKuOrag2svHwGeaPCOsyX+RcKuCIiD9wDfAHpVSpUmoz8BVwc1P7K6VeVErtUkrVKqUOAl8C0x3X4q4hKdMyfqNnqGnNmDQoCD9vD7cYxymtNF+Fedgg50zc4GC8PQ3mujzMN8H+7PUfeXF1OpU1xhaP3Xj4DJU1JuaP7BrjNwDiSlMZRWQ8sEUp5WdV9jAwWym14DzHCrALeEsp9aalLBPohjmw7gYeUUqlNHP8EmAJQEhISOyKFSs6/oZacKTISHqhkWG9PIgK8rDra7VXaWkpAQEBvJ9axY+nanntQj88DF3jm5i1uvPQlbXmHLyaXEl2qYkXZ3Vz2W/sSike2VhBoI8wLsSjzb9/TZ0H69/lvgEGPkmvZlN2Lf38hYsivThXrZp8nXf2VrE7r5a/X+CHpxv9XrX0WZgzZ06yUmpic8e62tI2AUBJo7JioHsrjl2KObD8y6psMeYgJMCDwBoRGaaU+sk1r1LqbeBtgJiYGBUfH9/Wtjdp48EzfLv/FH17diPAx5O8c1Wk55Sw+Ug+JgXenkY+vss1Fz1MTEwkPj6eF/ZsZNJgHy68YIqzm+QUdeehK2vNOTjhm8UfvthPxKhJDA5xzQCdllNC/ppNPHTxKBZNGdjm45s6D/GN9rlsHiQezOM3K1L4V2o1AB5Sy4Kx/Rgd3pOQ7j70DvAmJSGJyJAeBA0Z5ZK//83pyO+DQwOOiCQCs5vZvAX4JdCjUXkP4Nx56r0f81jOTKVUVV25UmqL1W7Pi8itwEzg67a1vH3+uzeHez/a1aDMx9OAr5ehPm95da2J/+w87rIfuOKKGg7mnuPS0X2d3RTNxcVHhwDm6dGuGnDWW26ynGvnNcviY0K5KW4gf//+CAowKsVXKdl80ehG0bScEhb/n33SHbgih47hKKXilVLSzGMGcAjwFJGhVoeNBZpdQ0JEbgceBy5USp0vG5TCfLVjd4Vl1Tzxxb765waBBy6IIv1PF7Pstsn4epn7fQVYuSub7/blOKJZbbbreBFKwUR9w6d2HgN6+TE4xJ9EF54evS4tl3EDAgntYf81y2ZFh+Jj+T339TLw6d1T2fPkPNb9ehbXTwyvy/RRv9J2V+BSkwaUUmXASuBpEfEXkenAlcAHTe0vIouB54B5SqmMRtsGish0EfEWEV/LlOnemK+k7Kqi2sjt7+2ktKoWbw/zB87b08DsmFBEhNiIIJbfGcdD82P4988nM3ZAIPd9tIuPdxy3d9PaLCmzEA+DMG5AoLObormB+OhQtmUUUFHd8oC5M5wurmTvyWLmjXDMPS/Wv+fL74wjNrIXgX7eDA3rzsJJA+uDUVfKLeVqYzgA9wLLgDygALhHKZUKICIzge+UUnXX688AwcBOq0HKD5VSv8A87vMGMASoBPYAlyil7PpVotZo4pcf7ybl5FneWBxLSHcftmUUEDc4uMElc2xEUP3ziYOCuHf5Ln67ch9F5dXcM3uIywy67swsYlS/Hvh5u+JHRXM18TEhLNtyjG3HCpjjYpkr16WZu9PmOyjgQMPf88bly++Ma/JvQ2fmcn9FlFKFwFXNbNuEeWJB3fNBLdSTCoyxeQNboJTiD1+msj4tl6evHFl/B/P5Pkx+3p68c8tEHv40hRdXHyQ9p4SYPt2JG9zbqR/EWpMi5cRZboqLcFobNPcyeVAvfL0M/HDwjOsFnAO5RAT7ERXqGuNLzQWjzszlAo47++eGI3y84zj3xg/hlqmRbTrWy8PAXxeOo7rWxFcpOUhKDj5eR5w6mJhZYqKq1qQX7NRazdfLg6mDg0k8mAc0d7+2452rrGHr0XxunRrpMr0HXZFLjeG4q+SsIu75MJm/rDvE1eP788hFMe2qx2AQRvU3T9JzhcHEw0Xm/O6xesFOrQ3iY0LJLCgnM7/M2U2pt/FQPjVG5bDxG61p+gqng5Kzirjx7W1UG00YBK6fNKBD36DiBvfGx/MIVbUmu+ROb4tDRUYG9fYnpHvbVtLVurb4GPP06B8OnSHSRZbcX3fgNEF+Xl2uC8vV6CucDkpIz6PaaL4SECCpAwsBgrlf96O74ojo5UdgNy+nzQ5TSnG4yKh/QbU2iwj2p29PX/79Y2aHFsa0lRqjiQ3peVwwLAxPD/0nz5n02e8ApRQ7M81dXgYbTm+MjQjit5cOI7+smg3peR2urz2OnimjtAY9fqO1WXJWEXnnqsjIL2PxO9ucHnR2HiukpLKWeSNcaxJDV6QDThOKq1SrfklWJJ1g+7EibpkawW/q5trb6Ipg7vAwwnr48OG2LJvU11bJWXrBTq19tmUU1KebrnKBmxrXpeXi7Wlg5tAQp7ZD0wGnSUVVikXn+WaWcaaUpV8dYNqQYJYuGMl9c6Js2v3k6WHghkkD2Xj4DMcLOpbCtj1W7z+NtwHOllU7/LU191a3gnIdZ14lK6VYdyCXGVG98ffRQ9bOpgNOM6pqTXyxO7vJbTVGE7/6zx68PQ28snAcBjut9Hrj5IEYRFi+w7FXOcmZhSQePEO1CRa/u93pXSKae6m7qfGqcf1QQH6p8760pJ8+x8miCj07zUXogNMMAT7eeZwVO0/8ZNur6w+x92QxL1w9mj497bcmU5+evswdHsqnSSepqnXcUiFf782pT1Dl7KnZmnuKjQjiLwvHERHsxzubMs5/gJ2ssyzWeaGdF+vUWkcHnCYE+QjLfj6JuEHBPPr5Xp5YtY/qWvNMtO0ZBbyeeJTrJw7gEgesoHxTXASFZdV8t++03V+rTl2aYANda50nzbY8DMIdMwax+/jZ+jFBR1tft1hnd/sv1qmdnw44TejpI8yJCeW9n0/i7tmDWb79ODe+s42v9mRz1/tJhHX34ckFIxzSlulDehMZ7OfQyQN7TxYzvG93rh7q1WWWTdfs49rYcHp28+L/Nh1z+GuvS81l78liRvRrnPFEcxYdcFrg6WHgt5cM57VFE9ifXcwDn+yhpLKWwvIa0k+3mKLHZgwGYfGUCJKyikg/3Tg3ne0dyy8j/fQ5rosdwOVDvHWw0TrEz9uTxVMGsib1tEMnvyRnFXHP8mQAPk8+qcchXYQOOK1w2Zi+LJr8v+yARqNjxzWujQ3H29PgkKucNanmrruLRnWdPOuafd06LRIPg7Bsi+OucrZlFFBryXJY6+DfV615OuC00uVj+9UnTXP0uEaQvzeXj+nLql3ZlFbV2vW1Vu8/zZjwnvQP7GbX19G6jrAeviwY248VSScoLq9xyGvWrdAh6HFIV6IDTiv9JJmSg7uaboqLoKza2OxUbVvIKa5gz4mzXDRSX91otnXnjMGUVxv5yEFJBuu+mC2cGK7HIV2IvhOqDZyZv2L8gEBG9O3BOxszKK6otkuunDX7zd1pl+juNM3GRvTrwfSoYN778Rh3zBjU4MZQe9iQlkd3H0/+dNVou7+W1nr6J+EmRIRZQ3uTVVjOX9YeYvH/2X6NqtWpp4kOC2BwiGskqNI6lztnDia3pIpv952y6+uYTIoNB/OYFROig42Lcbmfhoj0EpFVIlImIlkisqiFfZeKSI2IlFo9BlttHyciySJSbvl3nGPehX34ensAYFK2vyGzoLSKHccKuVh3p2l2MntoCFGhAfxt/WFeSzhst5lj+7KLOXOuiguH6Zs9XY3LBRzgNaAaCAMWA2+ISEupA/+jlAqwemQAiIg38CXwIRAE/Bv40lLulmYODcHLw7yMjsFg21w569NyMSk9O02zH4NBmDfcnJzNXlfpAN+n52EQcyI4zbW4VMAREX/gGuAPSqlSpdRm4Cvg5nZUF495jOpVpVSVUurvmCetXGCr9jqaeeLCFHr5e9M7wIcx4T1tVvfq/acZ0KsbI/rqm+Q0+/H1st9Vep0N6blMGBhEL3+3/W7ZabnapIFooFYpdciqLAWY3cIxC0SkEMgB/qmUesNSPhLYq+rWSTfbaylf3bgSEVkCLAEICQkhMTGx3W/C3m6JEV7dVckfP/yeuRFeHa6vvEax8VA58yI8+eGHH+rLS0tLXfo8OIo+D7Y7B/7njBjEHHAMAj5ns0hMPNnxBloUVZrYn13BtdFedvmZ6c9Cx86BqwWcAKDx7fTFQPdm9l8BvA3kAlOAz0XkrFLqY0tdxa2tSyn1tqUuYmJiVHx8fHva7xCzlWJH8Xa+zSrh0eun08O3Y0Hnyz3ZGNUe7rpkcoOZb4mJibjyeXAUfR5sdw7igdBB2TzwyR5umBzBnVeN6nCd1j7afhzYx5LLphId1tyfjfbTn4WOnQOHdqmJSKKIqGYem4FSoHGfTg+gyXVklFIHlFKnlFJGpdSPwN+Aay2b21SXOxERfnfpcM5W1PB6wtEO17d6/2lCu/sw3knprLWu5Ypx/ZkUGcSWI/k07IDouO/TcgkP6sbQUD3T0hU5NOAopeKVUtLMYwZwCPAUkaFWh40FUlv7EpjHabAcM0ZErJPVjGlDXS5tVP+e/Gx8f5ZtOcbJovavUVVRbSTx4BkuGtnHbnl9NK2xhRMHkJFfxs5M200aqKg2svlIPnOHh9Hw115zFS41aUApVQasBJ4WEX8RmQ5cCXzQ1P4icqWIBInZZOABzDPTABIBI/CAiPiIyP2W8g12fRMO9PD8GAR4ec3Bdtex8fAZKmqMXKxnp2kOdNmYvgT4ePLJTtutPLA1I5+qWhMX6OnQLsulAo7FvUA3IA/4GLhHKZUKICIzRaTUat8bgCOYu8neB/6slPo3gFKqGrgKuAU4C9wOXGUp7xT6BXbjzpmD+GLPKfaePNuuOlbvP02gnxeTB/Wyces0rXl+3p5cMa4f/92XQ0mlbdZXW5+Wh7+3B1MG68+yq3K5gKOUKlRKXaWU8ldKDVRKfWS1bZNSKsDq+Y1KqWDL/TfDLFOfrevarZSKVUp1U0pNUErtduR7cYRfzB5CsL83z36b1ub+8O0ZBXy7L4cJAwLx8nC5j4LWyV0/cQCVNSa+2tPxlQeUUmxIy2Pm0BB8PD1s0DrNHvRfGTfX3deLX82LZvuxQh5akdLqG+mSs4q46d3tVNea2HQkX+cL0RxuTHhPhvXpzn+aSOPeVqmnSjhdUskFOpW0S9MBpxOICQtAgFW7s7n+ra38eCS/xf0PnCrhNyv2UGM0XxGZTErnC9EcTkS4ftIA9mUXk3qq8R0MbbMhPQ8RmKNXF3BpOuB0Ajszi6iblFNrUvz8vZ28uDqdvJLKBvvlFFfw8KcpXPaPTZw5V4WnQZyS30fT6vxsfH+8PQ2s6OBVzvfpeYwNDySku4+NWqbZg6vd+Km1Q9zgYLw9DdTUmvDwMDBuQCBv/HCUdzZlcMXY/kyMDOLLPdnsyioChLtmDua++CiOnCllW0YBcYODdb4QzSkC/by5eGQfVu3O5reXDq9f+qYt8s5VknLiLL+ZF22HFmq2pANOJ1CXHM46eGQVlLFs8zE+3nGCz3eZlw4xCLy2aDyXjO5bf5wONJqzXT9pAF+lnGJN6mmuHNe/zccnpp8B4MLhYbZummZjOuB0njaWxQAACwtJREFUEo2DR0SwP3+8chQ9unnxzw1H6u+Izcgvc1obNa0pUwcHM6BXNz7ZcaJdAeez5BN09/Gkotq+6de1jtNjOJ1cfEwoPl4GPVajuSyDQVgYO4CtGQVkFbTtC9HWo/nsyCziXFUti9/drmdbujgdcDq5uu62h+bH6Nzumsu6dmI4Ajz2+d42BY2X1/5vYXl7pTvQbEcHnC4gNiKI++ZE6WCjuaxTZysRgW0Zha1OzPbR9uMkZxXhoWdbug09hqNpmtNtyyigbqGMyhoTmw6fafEL0o9H83nyy/3Mjg7hvjlD2JlZpGdbugEdcDRNc7q4wcH4eBmoqjGhgG/25nBzXATBAT+9r+ZYfhn3fLiLyN7+/GPReHr4ejF5kL6ycQe6S03TNKerG2t8+KIYHrs4hhOF5Vz75laOFzRMvVFcUcMd/96JQeDdWyd2OPmg5lg64Gia5hLqxhrviY/io7umUFRezdVvbGHfSfOyN7VGE/d/tIsTheW8eVMsEcH+Tm6x1lY64Gia5nJiI3rx2S+m4ePpwQ1vb+XdzRksfGsrmw7n8+xVo5miJwe4JR1wNE1zSVGhAay8dxq9A3z40zdp7Dp+Fg+DMESnj3ZbOuBomuaywnr4cuX4fv8rUHplc3emA46maS5tdnQovnq1jE7B5QKOiPQSkVUiUiYiWSKyqIV9vxORUqtHtYjss9qeKSIVVtvXOuZdaJpmK3q1jM7DFe/DeQ2oBsKAccC3IpKilEptvKNS6hLr5yKSCGxotNsCpdR6O7VV0zQH0Cubdw4udYUjIv7ANcAflFKlSqnNwFfAza04NhKYCbxvzzZqmqZp7eNSAQeIBmqVUoesylKAka049hZgk1Iqs1H5chE5IyJrRWSsjdqpaZqmtZGrdakFACWNyoqB7q049hbgmUZli4FdmFPBPAisEZFhSqmzjQ8WkSXAEoCQkBASExPb1vJOqLS0VJ8H9HkAfQ7q6PPQwXOglHLYA0gEVDOPzcB4oLzRMb8Bvj5PvTOAUiDgPPulYx7TabGd0dHRSlMqISHB2U1wCfo86HNQR5+Hls8BkKRa+Nvq0CscpVR8S9stYzieIjJUKXXYUjwW+MmEgUZuBVYqpUrP1wTMVzuapmmag7nUGI5SqgxYCTwtIv4iMh24EviguWNEpBuwEHivUflAEZn+/+3db4hcVx3G8e9jt83WbGJIUiO0mCW1qWZLamJFUIOBRmL7QtS8KRvFgBpsDYgFaUElWw0tguALaavBLa2JpEobI4L0RaERE4g0aGrdNqZEaLRGpTH/umnWP/x8cc6YcdzdJrOZc2f2Ph+4sPeeyfC7T4b7m7l79oykqyT1S/oysBjY37ETMDOzKXVVw8nuAq4G/gbsAu6MPCVa0hpJrZ9iPgacAp5pOT4PeBg4CbwCfAS4LSL8Z8pmZhXotkkDRMTfSU1ksrFfkiYWNB/bRWpMrY8dA1Z2okYzM7t03fgJx8zMZiE3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrwg3HzMyKcMMxM7Mi3HDMzKwINxwzMyvCDcfMzIpwwzEzsyLccMzMrAg3HDMzK8INx8zMinDDMTOzItxwzMysCDccMzMrousajqQtkg5KmpD06EU8/kuS/iLpjKRHJM1pGhuU9Iykc5IOS1rX0eLNzGxKXddwgD8D24BH3uiBktYD9wK3AkuBZcB9TQ/ZBfwGWAR8BXhC0jWXu2AzM3tjXddwImJ3ROwBTlzEwz8NjEbEWEScBL4BbAKQtBxYDWyNiNcj4kngeWBDZyo3M7Pp9FVdwAwNAT9t2n8OWCJpUR77Q0ScbRkfmuyJJG0GNufdCUm/60C9vWYx8GrVRXQB5+AMGpzD9Bksne4f9nrDGQBON+03fp43yVhj/NrJnigitgPbASQdjIhbLm+pvcc5JM7BGTQ4h5llUPSWmqS9kmKKbV8bT/kaML9pv/Hz2UnGGuNnMTOz4oo2nIhYGxGaYvtgG085BtzctH8z8NeIOJHHlkma1zI+1v4ZmJlZu7pu0oCkPkn9wBXAFZL6JU116+8HwGckrZC0APgq8ChARBwBDgFb83N8HFgJPHkRZWyf6XnMEs4hcQ7OoME5zCADRcTlLGTGJI0AW1sO3xcRI5LeDrwArIiIY/nxdwP3AFeTmsnnI2Iijw2SGtD7gGPAFyLi6c6fhZmZteq6hmNmZrNT191SMzOz2ckNx8zMinDDaSJpoaSfSBqX9LKk4aprKmG69esk3ZrXoTuX16Wb9g+7epWkOZJG8//7WUmHJN3WNF6XHHZKOp7XJjwi6bNNY7XIoJmkGySdl7Sz6dhwfp2MS9ojaWGVNXZS/lOW85Jey9vvm8YuOQc3nP/1IPAPYAmwEXhY0qQrE8wyk65fJ2kxsBv4GrAQOAj8qHh1ZfQBfwQ+BLyFNOPxx3kB2Drl8AAwGBHzgY8C2yS9p2YZNHsQeLaxk68H3wM+RbpOnAMeqqa0YrZExEDeboT2c/CkgUzSXOAkcFOeUo2kHcArEXFvpcUVImkbcF1EbMr7m4FNEfH+vD+XtKTFqog4XFmhhUj6LWkx2EXUMAdJNwJ7gS8CC6hZBpLuAD5Bmhn7joj4pKT7SQ15OD/meuBFYFHLMlqzgqS9wM6I+H7L8bZy8CecC5YD/2o0m2zKtddqYoiUAQARMQ4cpQaZSFpCek2MUbMcJD0k6RxwGDgO/Jz6ZTAf+Dpwd8tQaw5HSXdFlperrrgHJL0qab+ktflYWzm44VwwAJxpOXaatC5bXU21Ht2szkTSlcAPgcfyu/da5RARd5HObQ3pNtoENcuAtPL8aET8qeV43XK4h/S1L9eS/uDzZ/nTTFs5uOFc4LXX/l/tMpH0JmAH6d3alny4djlExL8jYh9wHXAnNcpA0ruBdcC3JxmuTQ4AEfGriDgbERMR8RiwH7idNnPo9dWiL6cjQJ+kGyLipXys7muvjZG+cwj4733765mlmUgSMEr6JejtEfHPPFSrHFr0ceFc65LBWmAQOJZeEgyQltlaATxF0/qNkpYBc0jXjzoIQLSsY3nROUSEt7wBj5O+JXQu8AHSR8ShqusqcN59QD9phtKO/HMfcE3OYEM+9k3gQNX1djCH7wIHgIGW47XIAXgrcAf5AgusB8ZJs9VqkUHO4c3A25q2bwFP5AyGSLfe1+TrxE7g8apr7lAOC/JroHE92JhfD8vbzaHyk+qmjTTdc08O9RgwXHVNhc57JL9zad5G8tg60i+PXyfNWBqsut4OZbA0n/d50u2CxraxLjnkC+ovgFP5YvI88Lmm8VmfwRS5jJBmajX2h/P1YZz0BZALq66xg6+HZ0m3yU7lN2MfnkkOnhZtZmZFeNKAmZkV4YZjZmZFuOGYmVkRbjhmZlaEG46ZmRXhhmNmZkW44ZiZWRFuOGY9QtJ8SSOS3lV1LWbtcMMx6x23AFuBK6suxKwdbjhmvWMV6asCXqi6ELN2eGkbsx4g6UXgnS2Hd0fEhirqMWuHG45ZD5D0XtJq5mPA/fnw8Yh4ubqqzC6Nvw/HrDc8R/oytO9ExIGqizFrh3+HY9YbhoCrgF9XXYhZu9xwzHrDatL39RyquhCzdrnhmPWGVcDRiDhTdSFm7XLDMesNK/B0aOtxnjRg1htOAaslrQdOAy9FxImKazK7JJ4WbdYDJN0EjAIrgX5gTUTsq7Yqs0vjhmNmZkX4dzhmZlaEG46ZmRXhhmNmZkW44ZiZWRFuOGZmVoQbjpmZFeGGY2ZmRbjhmJlZEf8BFH4ypS7eFV4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "y_pred = model.predict(X_valid)\n", "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "7Mk8hrHT5jk9" }, "source": [ "Since a SimpleRNN layer uses the tanh activation function by default, the predicted values must lie within the range –1 to 1. It might be preferable to\n", "replace the output layer with a Dense layer: it would run slightly faster, the accuracy would be roughly the same, and it would allow us to choose any output activation function we want. If you make this change, also make sure to remove `return_sequences=True` from the second (now last) recurrent layer" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "nT3gQCrq5jk9", "outputId": "2aa5a397-0474-4438-9733-360a6ecc9278", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_3\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " simple_rnn_4 (SimpleRNN) (None, None, 20) 440 \n", " \n", " simple_rnn_5 (SimpleRNN) (None, 20) 820 \n", " \n", " dense_1 (Dense) (None, 1) 21 \n", " \n", "=================================================================\n", "Total params: 1,281\n", "Trainable params: 1,281\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n", " keras.layers.SimpleRNN(20),\n", " keras.layers.Dense(1)\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": { "id": "IXZoG7QA5jk-" }, "source": [ "## Forecasting Several Steps Ahead" ] }, { "cell_type": "markdown", "source": [ "So far we have only predicted the value at the next time step, but we could just as easily have predicted the value several steps ahead by changing the targets appropriately (e.g., to predict 10 steps ahead, just change the targets\n", "to be the value 10 steps ahead instead of 1 step ahead). But what if we want to predict the next 10 values?\n", "\n", "The first option is to use the model we already trained, make it predict the next value, then add that value to the inputs (acting as if this predicted value had actually occurred), and use the model again to predict the following\n", "value, and so on, as in the following code:" ], "metadata": { "id": "cg0V6AMCJ8Zg" } }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "aEXPkSXD5jk-" }, "outputs": [], "source": [ "np.random.seed(43) # not 42, as it would give the first series in the train set\n", "\n", "series = generate_time_series(1, n_steps + 10)\n", "X_new, Y_new = series[:, :n_steps], series[:, n_steps:]\n", "X = X_new\n", "for step_ahead in range(10):\n", " y_pred_one = model.predict(X[:, step_ahead:])[:, np.newaxis, :]\n", " X = np.concatenate([X, y_pred_one], axis=1)\n", "\n", "Y_pred = X[:, n_steps:]" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "2QT45Nd85jk-", "outputId": "4b717995-5a87-43c0-8814-4c06a28b59a3", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1, 10, 1)" ] }, "metadata": {}, "execution_count": 26 } ], "source": [ "Y_pred.shape" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "sJ_ocxoY5jk-", "outputId": "c614333d-4fdd-41fe-97ea-a9d5fe9577a4", "colab": { "base_uri": "https://localhost:8080/", "height": 314 } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Saving figure forecast_ahead_plot\n" ] }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVzVVf748df7soqgAuKGAuK+5L47KpjtY1qWU5ItU9q0TXvNTDUt35omp5nf1GSL6bSMW5aVVraZkEsiivuSGwIqbggomyz3nt8fn3sVkZ27fIDzfDx4AJ/tno8I73vO533eR5RSaJqmaZrZWDzdAE3TNE2riA5QmqZpminpAKVpmqaZkg5QmqZpminpAKVpmqaZkg5QmqZpminpAKVpmqaZkikDlIg8KCKbRKRIRD6s5thHReS4iJwVkf+KiF+ZfVEiEi8iBSLyq4hMcHnjNU3TNKcwZYACMoCXgf9WdZCIXAX8CbgciASigRfLHLII2AKEAs8An4lImCsarGmapjmXmLmShIi8DHRUSt1Zyf6FQKpS6i/27y8HFiil2olId2AH0FoplWvfv8a+/1233ICmaZpWZ96ebkA99QGWlfl+G9BWRELt+1IcwanM/j4VXUhEZgIzAfz9/QdHRES4psUmYbPZsFjM2oF2Dn2PDV9jvz9oGve4b9++TKVUrUevGnqACgTOlPne8XVQBfsc+8MrupBSag4wB6BHjx5q7969zm2pySQkJBATE+PpZriUvseGr7HfHzSNexSRtLqc19DDdh7Qosz3jq9zK9jn2J+LpmmaZnoNPUDtAvqX+b4/cEIpddq+L1pEgsrt3+XG9mmapml1ZMoAJSLeIuIPeAFeIuIvIhUNR34M3C0ivUWkFfAs8CGAUmofsBV43n7+DUA/YKlbbkLTNE2rF1MGKIxAU4iRQn6b/etnRSRCRPJEJAJAKfUdMAuIB9KBNOD5Mte5BRgCZAN/B25SSp1y211omqZpdWbKJAml1AvAC5XsDix37L+Af1VynVQgxnkt0zRN09zFrD0oTdM0rYnTAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJR2gNE3TNFPSAUrTNE0zJdMGKBEJEZEvRCRfRNJEZFolx30rInllPopFZEeZ/akiUlhm/w/uuwtN0zStrrw93YAqzAaKgbbAAOAbEdmmlNpV9iCl1DVlvxeRBGBVuWtNVEqtdGFbNU3TNCczZQ9KRJoDU4DnlFJ5Sqm1wHJgejXnRQFjgI9d3UbNdZLTspkdf4DktGxPN0XTNA8SpZSn23AJERkIrFNKBZTZ9gQwTik1sYrz/gqMV0rFlNmWCjTDCMZbgCeVUtsqOHcmMBMgLCxs8JIlS5xzMyaVl5dHYGCgp5txid2ZpbyeXIRS4GOBp4b60zXYq07XMus9OlNjv8fGfn/QNO4xNjY2WSk1pLbnmXWILxA4W27bGSComvNuB14uty0O2AwI8DDwvYj0VErllD1IKTUHmAPQo0cPFRMTU7eWNxAJCQmY7R4PnMzlsfcSsdnfM1kVFLWKJCama52uZ8Z7dLbGfo+N/f6gadxjXZlyiA/IA1qU29YCyK3sBBH5DdAO+KzsdqXUOqVUoVKqQCn1KpCDMQyomcjX2zOY9NY6Sq02vCwCgMUijIgO9XDLNE3zFLP2oPYB3iLSTSm1376tP7CrinPuAD5XSuVVc22F0ZvSPCw5LZtfDmay91guX+84xqCIVrwdN5j0rHxmfJxMWKAvgyJaebqZmqZ5iCkDlFIqX0Q+B14SkXswsvgmAaMqOl5EmgFTgRvKbY8AOgEbMXqLDwGtgXWua71WE8lp2Ux7P5GiUhsA117Wjn//biC+3hbatfTn6at78pcvdrD2QCZjuoV5uLWapnmCWYf4AO7HSG44CSwC7lNK7RKRMSJSvpc0GWPoLr7c9iDgHSAbOApcDVyjlDrt0pZr1UpMyTwfnCwCfTq0xNf7wn/HKYPDadfCn7dWHfBUEzVN8zBT9qAAlFJZGIGn/PY1GEkUZbctwghi5Y/dBfRzVRu1urOIMcoqgK+35ZJnTX7eXswcG81LX+9mU2oWQ6JCPNBKTdM8ycw9KK2RKrXa+Cz5CB1a+fPYld1ZcM8IBkcGX3LcrcMiCG3uy1vxuhelNR0LFkBUFFgsxucFCzzdIs/RAUpzu0+Tj3DwVD5//W0fHhrfrcLgBNDM14vf/6YzCXtPsfPoGTe3UtPcb8ECmDkT0tJAKePzzJnG9qYYuHSA0tyqoLiUf/24j8GRwVzVp221x08fGUmQvzezdS9Ka2QcAWf8+HHnA84zz0BBwcXHFRTA/ffDPfcYAesJNYvOafHnA1eF4uNh1iwX34Hr6QCludW8NYc4lVvEX67tiUj12f4t/H24c1QU3+48zv4TlU6D07QG5eKekpCWBnfcYXxfkbNn4dw54+uNDGUJUxlWEM8zz1RwcHw8TJ0KQ4e6rP3uogOU5jaZeUW8tzqFK3u3ZXBkzZMe7hrdGV8vCw8v3qrr82mNQkU9Jau1ZucmEMtUlrCEqUSnlUtcdgSnJUsgNtY5jfUgHaA0t/nPT/spLLHy1NU9a3Xeocx8rEqx+9hZbn0/UQcprcFLT698X0DAha+fZBZX+8UTWq6giiNIfWqZCv/6F7z2WqMLTqADlOYmqZn5LNiQzu+GdqJrm9oVxkxMOY2jqHFxqY0fdx93RRM1zW0iIireHhkJc+YYn0XgcNuhfOE7lcX3xl8UuAA2NIvlyPX3weOPw0svNbrgBDpAaW6QnJbNvfOT8bIIj0zoVuvzR0SH4uttwV6ijxXbj3P2XImTW6lp7vPKK1wScAICjO1xcZCaCjYbLDoei/+yJUyYM5Vlj8SfD1ygWBj+BP2X2WtjFxTANdc0quAEOkBpLpacls2t7yey93guVpvicFZhra8xODKYBfeM4PEre/DCxN5knCnkvvnJFNsrUWhaQxMXV7anpM73nOLiKjg4NhaWGEEq9YN4bAcPcaDVECYf+CelkdHQyl6vculSY5ivEdEBSnOpxJTTlNgDiVKKxJS6VZkaHBnMA7FduXN0Z16b0o91B07z5893YMb1zDStJhw9pVWrfiY1tZLg5GAPUlx/PfToQXTOFj7lJs6dOAOffw5DhkCXLsYwXyMKUjpAaS41IjoURza5TwUljepiyuCOPDqhO0s3H+Hppdv16rtawzFrVvUBpLI5TDYb5OVBSQlyw2Su9k/gxpIlpHeJNQLYr7/Cxx83qiClA5TmUv06tsTbYmFQRKtKSxrVxR8v78r4HmEs2XSE17/fS9xcnd2nNQBDh1YdQKqaw/Tss8YDqGnT4MsvKXniz6xSsfTuDdf9IwZKSvhptY/R02okQUoHKM2ldh49Q7HVxj1jop0WnABEhIH26ymgpNRW5+FDTXMbx1BdRQGkqjTx+fMhMRFuuw3694fXXyfgjVeJUfHk58NqxlCKF5teT2BBhv01Nm503325iGmrmWuNw8bULACGuqAa+agurXnTaz8lVoWXl3OGDzXN5coGqSVLjF5RVcEpPh5mzAAvL2O+U/v2ANw1ayCLcqcylSUkEMsmhvCb0njinoG41NhGkdGne1CaSyUdyiK6dXPCgvycfu3BkcG8NW0QAHHDI5zaQ9M0l3IEqZtu4rKnn646ON18sxHEbrnlfHAC+OTkhYoSMcQTTyzDSOJ0WnWLijccOkBpLmOzKZIOZTGss+vWcrqqTzs6hTTjWM45l72GprlEbCx0705oUhJcdVXFPZ6NG43AVFgIDz980a6IiAsVJYaykQRi8KGUyW1+cdMNuJ4OUJrL7D2Ry9lzpS4NUADDokJJSs3SKedawxIfDwcOUNyyJSxaBMuWXXrM44/Dd9/ByJGXJE44JvsmEMs/eIp1jKYEb/40vOEnRzjoAKW5TNIh4/mTqwPU8OgQsvKLOXCy8QxtaI3PptSsC1Miyjxz2u5IKf/d7y5NnFixAg4evKT3BBdP9gXIJ5CsLsPoc1IHKJcTkRAR+UJE8kUkTUSmVXLcCyJSIiJ5ZT6iy+wfICLJIlJg/zzAfXfRtCUdyiK8VTM6BgdUf3A9DLcHwER7QNQ0s1m5+wQ3v7uef/6wlzeffY+Sm24+/8wpr3t3+L//g6IimDTp4iD1xhsQHg433ljhdR2TfX/+2fj+dP9Y2LQJchvH0jSmDVDAbKAYaAvEAe+ISJ9Kjv1EKRVY5iMFQER8gWXAfCAY+AhYZt+uuZBSig2Hshga5frEhYiQANq28DvfY9M0s3lv9UEUMDx1O/9a+irfPPPvi585WSzQu7cxGfemm4wgtXMn/PQTPPAA+PhUuQjhqFEQEgLLz8QY63asXeuW+3I1UwYoEWkOTAGeU0rlKaXWAsuB6bW8VAxGKv2/lVJFSqk3AQHGO7O92qVSTxeQmVfEsM6uT/0WEYZ3DiXp0Gn9HEoznZyCYrYfOYNFYMDxfTw25c90mnLdxQcNHw7HjhnBJTISkpLgzTfB399Y2bCaRQi9veG3v4X/JI9COYJZI2DWeVDdgVKl1L4y27YB4yo5fqKIZAHHgLeUUu/Yt/cBtquL/2ptt2//ruwFRGQmMBMgLCyMhISEet+EmeXl5bn0Hn8+Yq82fuoACQkpLnsdh1YlJZw4W8ySFfG0bW6873L1PZpBY7/HxnB/yw4UU1RqY8ZlvpzqeitjQrzIPbSNhEPG/ry8PBICA2n13HP0feYZvLdsIfWyy+i0eDEnrriCkx9/TO8XX2T388+TIwKV/Ht06dKaj3P6cjS6Ly2WL2fztde67yZdRSllug9gDHC83LYZQEIFx/YGOgBewCiMIHWrfd9zwOJyxy8AXqjq9bt3764au/j4eJde/9FPtqhBL/2gbDabS1/HYd/xsyry6a/VJ0np57e5+h7NoLHfY0O/v7xzJar/i9+ruz9MqvSYi+7xp5+U8vVVCoyPefOUat1aqVWrqn2t3Fyl/PyU+m74c0pZLErl5DjhDpwD2KTqEAtMOcQH5AEtym1rAVzy5E8ptVsplaGUsiqlfgHeAG6q7XU053LMfxJHpVgX69omkJDmvmzQz6E0E1mUlE5OQQn3xXSt2Qnjx8PChcbE3OhoePrpGi9CGBgIl18OH6XFGs+y1qypZ+s9z6wBah/gLSJlV7frD+yqwbkK4zkT9uP7ycV/JfvV8DpaHR3NKeRIdqHL08vLEhGGRYWw4ZCux6eZQ1GplblrDjEiOqR2VU6mTIHHHoOUFLjvvpqXLJo1iz/0jOfz4yOx+fpV/ByqikQLMzJlgFJK5QOfAy+JSHMRGQ1MAv5X/lgRmSQiwWIYBvwRI3MPIAGwAn8UET8RedC+fZXLb6IJ23jIdfX3qjKscwhHsgs5mlP7RRE1zdm+2HyU42fPcX9Ne08O8fHw0Ufw3HPwzjs1T3gYOpRrP5zKSNaTHj7y0mdVZRMt6rPshxuZMkDZ3Q80A04Ci4D7lFK7RGSMiJSdkXkLcABj2O5j4DWl1EcASqliYDJwO5AD/B6YbN+uuciGQ1kE+XnTq3350VXXGh5tBMSNephP8zCrTfHe6hT6hrdgTLfWNT+xbNHYl16q3dIZsbF4fbaEz72nsuNsJGzZAtnZl143NrZ+y364kWkDlFIqSyk1WSnVXCkVoZRaaN++RikVWOa4W5VSocqY/9RTGankZa+zRSk1WCnVTCk1SCm1xd330tQkHTrNkKhgvCzuef7k0LNdC4L8vfUwn+Zx3+48xqHMfB6I6Vrz57AVVTSvanmOisTGsuKOJYw5/aWRZrF6de2vW1VldTczbYDSGqbMvCIOnsp3y/yn8rwsjudQugeleY5SitnxB4kOa85VfdrV7KSqgkItg9SAR2OZyicogL/9rebXVcpUwQl0gNKcbFOqe+rvVWZY5xBSTuVzKrfII6+vaXPXHmLPsbNc07cdlpqOImzcWHVQcASTGixC2Ls3pHS5imPNoo0Jv3/4Q/XXnTQJgoKMKhYmCU6gA5TmZF9ty8DbIpRabR55fUdg1GWPNE9ITsvm1RV7AJi39pBRGLYmnnqq+qAQG2scVw0RmBIST2DhKQDOvPIWK5+poucVGwstW0J+vvE5JqZmbXYDHaA0p0lOy2bFjuOU2hR3fJBU819OJ+ob3pIAXy+S9HMozQO+3paBzV63pqTURmKK+/8frnwmnic2TuVWFnIOP1aqWPr/bWrlQWr+fDhyBNq2hUOHjLlXJqEDlOY0P+w+jqOmlKd+OX28LAyODNbPoTSPOJFrLJzpJeDjbWFEtJufxcbHM/DvxjLwK/gtXzGR0fzCrSxi4N8rSYi4917j66Qk6NcPXn/dGOYzAR2gNKc5ddZ47mPx1C+n3bCoEPaeyCWvWBeO1dznXImVtfszGd21NY9d2YMF94yo3QTd+rInONxsW0ICxnDhQqbRjhMIiptt5RItHAkRrVoZw3wREcbKvl5eMH06rKpkuqgb50fpAKU5xbkSKwn7TjEsKpjHPfHLWcbw6FCUgoW/FnlkmFFrmlbsOMbZc6U8ENuFB2K7uv//vz3RIiXywrOsb7mGHFoyjYXGdkeihSM4Pf88ZGTA7bcbJ1xzDfj5QXExTJ5ceQq6m+ZH6QClOcWKHcfIyi/mj5d398wvZxlWm5Gg8UuGlbi5iTpIaW6xKCmdqNAARnpo5MCRaOFYCh6gCH+WMoUpLOXvzxdeSLRwZA3u3g3NmhnllcDY/+WXxvodJSUX96I8kIKuA5TmFB+tTyM6rDmju3rol7OMzek557/21LMwrWnZfyKXjanZ3Doswm0FkitTfin4z3ym0YJcbmmx4sJBTz1lrHK4eLHRUwoKurBvwgSYOxfOnTOW6vXg/CgdoLR623Y4h22Hc7hjZJTHfzkBRkSH4uNltMPL4rlnYVrTsSjpMD5ewk2DO3q6KcCFpeAfeADW+cSg2rUzqqSXtWKFUQppegXrwN5xB8yYYVREv/lmj03e1QFKq7eP16fR3NeLGweFe7opAAyODObDu4bhJTC+ZxuPDjdqjd+5EitLNx/hqj7tCA3083RzLjJhAuQWeHH0N7+Db76BnAujC/zvf0Zq+RVXVHzy229DWBgsXVq7qupOpAOUVi+n84r4ansGNw7qSJC/j6ebc97orq0Z3NaLjalZHps0rDUN3+08zpnCEqYNi/B0Uy4RGwsWC3wdNA2KiuCLL4wdWVnw9dcwbZrxvKkia9ZAbi74+tauqroT6QCl1csnmw5TXGrj9pGRnm7KJYa39+Z0fjHr9TMozYUW2pMjzDiU3LIlvNFxFjvW50GXLheG+ZYsMZIgHMN75VPHHc+cZs40Mvpmz655wVon0gFKqzOrTbEgMZ1RXULp1jao+hPc7LLWXgT6efPVtgxPN0VrpA6czCPpUBa3DIuoed09N2s+bigv/Po7ioeMMrLyjh2Djz+GPn1gwIBLU8fLJkRMmmRsCwmpXVV1J9EBSquzn/ac4GhOIbePjPJ0Uyrk6yVc2act3+08TlGp1dPN0RqhxUnppkqOqEiXe2KZyhJjSM9mMyqcr19v9J4SEi5OgCifrde7t3GR3btrv/SHE+gApdXZx+vT6NDSnwm92ni6KZWa2L8DZ8+VsmZfpqebojUy6w9mMn9DGkMjQ2htsuSIskaMgI3NY3l7wlLjedNbbxkVZTt3vjQ7r3xV9bZtjUoTe4wCuLWpqu4MOkBpdbJ861HWHsgkpmcbvL3M+9/oN11b0yrAh6+262E+zXmS07K5/b9JnCuxsSkt29STwX19Ydw4eGdPLNx9t7ExMtLIQS+fOl6+qrqI0YvavfvCthpWVXcG0/5lEZEQEflCRPJFJE1EplVy3JMislNEckXkkIg8WW5/qogUikie/eMH99xB45Wcls2jS7YBsDT5iKl/OX28LFzTtz0/7j5BYbEe5tOcY/nWDEqsRq1Hq838k8GvuAL27YOjv3/OiFipqTVPHe/V60IPys1MG6CA2UAx0BaIA94RkT4VHCfA7UAwcDXwoIjcUu6YifYl4QOVUle6stFNwc97T2K1rylQajX/L+fE/u0pKLay6teTnm6K1ggczipg2bajgOcLI9fUhAnG522f7YMWLeC552qeOt67N5w6BZnuHyY3ZYASkebAFOA5pVSeUmotsBy4ZMqzUmqWUmqzUqpUKbUXWAaMdm+Lm5YzhSVAw/nlHN45lLAgP53Np9VbZl4R0+dtQCl443cDPF4Yuab69IEbg+P5zZv2Z04vvVTzhIdevYzPHuhFiVLmW5JARAYC65RSAWW2PQGMU0pNrOI8ATYD7yml3rVvSwWaYQTjLcCTSqltFZw7E5gJEBYWNniJSdZDcZW8vDwCAwNrfZ7VpnhqdSHNfWBYO296hnjRNdjLBS2sv7L3uGBPEfGHS/nP+ACaeZszHbgu6vpzbCjMdH+FpYpZSec4mmfjyaH+dHPS/3t33GOrLVuIevplpvsu4tHlvlgsF7b3fvFFdj//PDkDB1Z4rv/x44y49Vb2PvYYxyZW+ue3SrGxsclKqSG1PlEpZboPYAxwvNy2GUBCNee9CGwD/MpsG40RoAKAPwPHgVZVXad79+6qsYuPj6/TeV9vy1CRT3+tvt95zLkNcoGy97gpNUtFPv21Wpp82HMNcoG6/hwbitrc36bULPXWqv1qU2qW09uReDBTjZu1SnX+09fqx13HnXptl/8MV61SqnVr9d3TqxQotXVrxfvVqlUVn2+1KhUQoNTDD9e5CcAmVYdYYMohPiAPaFFuWwsgt7ITRORBjGdR1ymlihzblVLrlFKFSqkCpdSrQA5GANTqYN7aFCJDA7i8V1tPN6VWBkW0IrxVM+YnpjE7/oCpEzu02ktOzeJ3763nnz/sdfoSK2v2n+LW9xNJPV2Al0UIbu7rtGu7hT11vO9DRkLEjz+W219d6rjFAj17emSIr5IiTB63D/AWkW5Kqf32bf2BXRUdLCK/B/4EjFVKHanm2gojsUKrpS3p2WxOz+H5ib3xMums+cqICEOiglm2NYOth3Pw9bY0iGcHWs188EsqpfbEHccSK3X52SanZZOYcpqe7YI4cbaI73cdZ83+U9gvjc2m6nxtj7GnhIcDHToY+RFPPWUsoPvKK0blc2Jjq87o693bmNTrZqYMUEqpfBH5HHhJRO4BBgCTgFHljxWROOBvQKxSKqXcvgigE7AR4xnUQ0BrYJ1r76Bx+u+6VIL8vLl5SCdPN6VOHMVsbap+f8Q0c8kpKGbN/lMIxrtPROqUuJOcls0tc9afTx8HiAwNYGL/Dny78zhWq61BJAVVZsECOHkSSkuN79PSjFJ7YA9SVenVC+bPh7NnjSxANzHrEB/A/RjPjk4Ci4D7lFK7RGSMiOSVOe5lIBTYWGau07v2fUHAO0A2cBQjDf0apZS586JNKCOnkBU7jnHLsE4E+pnyfU21ru/fHjC6zw35D412sde++5W8Iiv/uLkfAyNaYbUpvOvQw5+7JuV8cBLgzlFRJDwRwxu3DGTRjBE81kAy9irzzDMXgpNDQYGxvVqOkke//ur0dlXFtH9plFJZwOQKtq8BAst837mKa+wC+rmkgU3MR+tTUUpxx6goTzelzoZ1DmVgRCvSThfw/u1DGuwfGu2CTalZLEo6zIwxnblpcCeu7NOOK/71M3/+fAfLHxxd4yonG1Oz+GH3cSxy4Q3MxP4dzi/AOTgyuMH/f0lPr932i5RNNR82zGltqo6Ze1CaSeQXlbJoQzpX921Hx+CA6k8wsSt6tyUrv5jI0IZ9HxoUl9r4yxc76NDSn0cmdAeghb8PL0zsw+5jZ/nvukM1uk766QLu/V8ykSHN+fDOYQ2+p1SZiEqWq6ps+0W6dAEfn4tLHrmBDlBatZZuPsLZc6Xc/ZtKO6sNxvDOxrDexkNZHm6JVl9z16aw70QeL03qS/Myw85X923HhF5t+H8/7udwVkGV18g9V8LdH23EalPMvWMIY3uE8UBs10YXnMBIiAgo974sIMDYXi1vb+je3e2ZfDpAaVXalJrF//txH13bBDIoouH/0l4W3pJmPl5s0AGqQUs/XcAbK/dzdZ92TOh98ZQHEeHFSX0Rgb8u2+mYD3kJq03x0KItHMrM5524QUSHmWNCsKvExcGcOcYq7mAUKp8zpwYJEg7li8a6gQ5QWqWS07K59f1EsgtKSDudz+b0HE83qd58vS0MimylA1QDlpyaxe0fJGER4YXrKyrPCeGtmvH4lT2I33uKPy7ecsm8qOS0bKa+t56Evad4cVIfRnVt7Y6me1xc3IUY8/DDtQhOYDyHOnQIzp1zSdsqYtokCc3zElMyz2c1Ncj5H5UY3jmU/7dyH2cKSmgZ4OPp5mi1kJyWzS3vJ1JiNTL1juYU0q6lf4XHXhbeAhH4atsxvtl+jBGdQwn09+Z0fjFb0rOxKfCyCD3buS9t2gxatzZq861eDX/+cy1O7N3bWPBw3z7o557cM92D0irlbS/Y1djSsod1DkEpI3NLazjyi0p58atd5980KaWqrKS/MTX7/Ix8m4JfT5wlPauAI9kF5yfeUs01GquxY2HdukvTzqvkyORz4zCf0wOUiLwpIl9XsL2FiLwgIr3KbHtERHaIiA6UJqOU4qvtGbQN8uOxK7s3qqymAZ1a4etlIUkHqAZjS3o21725hu1HzuBlEbxqUEl/RHQovt4WvAT8fSy8f/tQvntkLG/HDcbfx1KjazRWY8dCbi5su6RsdhW6dzfKHrkxUcKpQ3wi0gX4AxVUfACGAM8Dn5fZ9h5GiaI7gA+c2RatflbuOcnOo2f5x039GmzliMr4+3gxoFMrNjTBd84NSXJaNssPFLMicxtLNx+lXQt/Fs8cgY+XhcSU04yIDq3yTdPgyGAW3DPikmMr296UjLFXI129GgYPruFJ/v4QHX2hBzVrFgwdWnWJpPj4ei0P7+xnUI8A25RSmyrYNxAoAs73D5VShSLyMfAEOkCZhlKKf6/cR2RoADcMDPd0c1xieC9wNnYAACAASURBVHQIbyccJK+otMFWxmjMktOymfZ+IkWlNuAI47q35j/TBtHCXq6qpkGlsgm2jWHibX2EhxtTm1avhkcfrcWJvXtf6EENHWqsJ1V+2XiH+PgL++uoRkNrItJVREpE5KVy29+xL7U+RET8gNuAhRWcvwd4HfADSkREichS++7FQG8RqajXpZWTnJbt8mrcP+4+wa6Mszw0vluNZ+I3NMM6h2C1KV3V3KQSU05TXGoDjIUxh3UOPR+cNOcYOxbWrDHyHmqsVy8jSaK09EIV9IoWPSwbnGqyrHwlavTXRyl1AJgLPCIioQAi8lfg98AN9h7TCKAVsKaCS9wOpABfASPtH4/Z923FWEbj6jrfRRORnJbNrXMS672kQHJaNl8fLK7wfKP3tJ+o0AAmD+hQ3yab1uDIYLwtoof5TGpQRCsceQy+TfQ5kauNHQunT9fykVLv3lBSAgcPGt9XFKScFJygdkkSLwFewJ/sFcafB6YrpVba94/AKCa8vYJztwEdgVVKqUT7RxqAUspm3z+ijvfQZHy78xjFVhs2ZZR5qUv2kaNi82f7S5j2/qVB7ofdJ9h9rHH3ngACfL3pG96SJD0fypSOZBcCMDbcu1El6JjJ2LHG59Wra3FSRZl8jiB1880webLTghPUIkAppY4B/8ZYsuJd4I9KqbKDix2As0qp4gpO7wP4YizHXpFT9vO1SthsinUHMi98r6C5b+2XnP5qW8b5NN2iUhtvrNzPuRIrYPSe3li5n86tmzOpEfeeHIZHh7DtSA6FxVZPN0UrQynFR+tT6d42kLv6+urg5CKdOxvPomoVoHr2ND6X73aNGgUtW8KyZTBlilOCE9Q+zXw/xnOk9Uqp2eX2+WMkQVRkEEbvamsl+wsxltbQKrEgKZ09x3K5Pyaah8Z3JTIkgFnf763VMJ+1TJATjLH91ftPEft6Ap9uOszb8QfYfewsE/u1b9S9J4fhnUMosSq2HNbPocxkc3oOO4+e5faRUeeriWvOJ2L0olavhkqqQV0wa5YxdBcUBJ06XdyDKi2FCRMgJcXoYS1deukzqTqq8V8hEbkcIy18PTBaRMpPJT6N8QyqIgOBg0qps5XsDwEyK9nX5B3NKeTvK/bwm66tefKqnjx+ZQ8+vW8kYUF+3PVBEnuOVfbPerH/rU9l/8k8HpnQjSndfPj0D6NYPHMEbYL8ePKz7fzjh30AzFmT0iSSB4ZEhSACG1L0MJ+ZfPRLKkH+3o02g9RMxo6FjAwjtlTJkbEXH28EIUcPymaDa66BtWuNNPTZsytPnKiDmmbxDQK+wEiUiAHSgVfLHfYr4CsiHSu4RG/KpJdXoDOwtyZtaWqUUjzzxQ5sCl698bLz7yjbBPkz/+7hBPh6M31eEl9vz6gyuy8jp5B/fL+Xcd3DePjybvy2izF0MiI6lC8fGM1E+2J+cGG12cauhb8Pvdu30M+hTOTk2XOs2HGMmwd3uqhCueYaNX4OVTYZIijICFA2G9xwA6xcCX5+sGLFhaXjnRSkqg1QItIV+Bb4AXjI/ozpReBaERlb5lDHLVa0mlUO0F9ErhKREY5MQPv1WwHdy5yvlfHl1qMk7D3Fk1f1oFPIxbXyO4UEMP+e4RSVWnlo4ZZKs/uUUvx12U5sCl6e3PeSYRMR4c5RnZvk7PrhnUPZnJ5NUal+DmUGC5PSKbUppo+M9HRTmoRevYzafDV6DuUIPN9/D4WFcOWVsHw5+PpeCE7lj61nkKoyQIlIO4zAtAeIs2fcAXyM0WP6u+NYpVQqkARMrOBSfwVOAF9iDBH2KrPvOqAYo4emlXEqt4gXv9rNoIhWla5k27VNIDcMDEdhJE4UldhYf/Di0dJvdx5n5Z6TPHZF90uCnINjdn1jXaytMsM6h1BUamPHkTOebkqTV1xqY8GGdGJ6hNG5dXNPN6dJEDGqStQ4USI29sICUj/9ZCxi+O23MH58xccuWVKvShJVBiil1HGlVLRSKkYpVVRmu1Up1UspVX5y7TvAjSISUO46O5VSw5VSzZRSopRaW2b3bcCnSqmLxpREJEREvhCRfBFJE5FpFbVRDK+JyGn7x2tSposgIgNEJFlECuyfB1T9T2IOyWnZ3DZvA3nnSpl1Uz+8LJU/LJ40IBw/b+NHqYDl2zI4cDIPgDMFJTy/fBd9w1tw1+ioKl9zcGRwo12srTLDOocA8MZP+5vEczcz+27XcU7lFnHHyChPN6VJGTvWeAZ15EgNT7j9dvCyZxA//XTFwckhNhaeeqrObXN2qtZ8IAO4vyYH24PFeIwhw/JmY/Ss2gJxwDsiUtHiLzOByUB/oB9GD+5e+/V9gWX2dgUDHwHL7NtNyzFXae/xXADOFFZdcnhwZDALZ4zgyau68/DlXTlxtohr31zDs1/s5LZ5GzidV8Tfb+zXJDLzautQZj4CrNmfWa/Jz1r9ffRLKpGhAYzrHubppjQpjudQayoqsVCRLVugVSt49ll4912nZexVxKl/sZRSpcBdQNXrLF/QDrjTXqniPBFpDkwBnlNK5dl7XMuB6RVc4w7gn0qpI0qpo8A/gTvt+2Iw6g3+WylVpJR6EyPDuoqQ73kLN6TVeEkBB6P3041Hr+jBj4+NZWCnVszfkMaOo2cQEXtNM628sv+250psLNty1IOtabp2Hj1Dclo200dEYqlitEBzvl27jKG+adMgKgoWLKjiYEeViE8/hf/7P6dm7FVEKlsO2ZNEZCCwTikVUGbbE8A4pdTEcseeAa5USm2wfz8EiFdKBYnIo/Z915Q5/mv7/n+Wu85MjN4YAQEBgwsKahpjncuvU1/aTH0JsXgDCmUt5cTiZyjO+LVW12kx4mZajZmOWCwoWyk5axZwNvFT1zS6AfPt0JO2t7yCePmACCJCwf5EzqxbBF4++Edcxrn0HbX+99dqJ2zKX/GPGsjJpS9SlFrZdEnN+W4F3gfKPvPLB2YAiy46MgZYAkwFEmqwvZxkpdSQ2rbOrHmcgUD5yT1ngKBKjj1T7rhA+3Oo8vsqvY5Sag4wB6BHjx5q7173Z71vPZxD3PuJdGjVjL9c14vdGWeN5QD+Wfv1V5LTsombm0hJqQ0fP18+X/wegyMvFP5ISEggJibGia03n5reY3JaNokpp+nXsSVb0nOY6+9NQLcRON7I+3pbTJs40hh+jmv2n2L6vCQAom7720X/1o3h/qrjyXuMioK0tPJbmxMZuZDU1DJ1v8vU14uvpHJ5fBUljuo64dqsASoPKL8OcwuMorLVHdsCyFNKKRGpzXU8au/xXO74bxIhgb7Mv2c4bVv4E9ujTZ2vp9e8qbmySy+M6RbGHaOiuG9+Mr8cNIb/HPPC9L+hayzckH7+a/1v7V7p6TXcvnFj1fX1ymbsOanMEZh3yfd9gLeIdCuzrT+wq4Jjd9n3VXTcLqCfXBy++1VyHY/5ZnsGN769DovAgrtH0LaFv1Ou2xSz8pyhZTMfHr+yB972LpSXV9OZF+YJGTmFCDS5OXhmEBFRw+1PPVV94Klnxl5FTBmglFL5GCvvviQizUVkNDAJ+F8Fh38MPCYi4SLSAXgc+NC+LwGwAn8UET8RedC+fZUr218bvxzI5MGFW8gvtlJQbOVUXmXlDDV3GhwZzAd3DcXXy8KQJr64nSudyi1ix9Ez3DgovMnNwTODV16BgHJTI/39L0x18jRTBii7+zEKyJ7EeFp3n1Jql4iMsQ/dObyHsc7UDmAn8I19G/aqF5Mx1qPKwVi/anIlFdc9YumWI+fXvSm1No0SQw3FmG5h3Dk6isSU0xzO8kzSTGO3YscxbAruHddF9/Y9IC4O5syByEgjkw+Muq9xcZ5tl4NpA5RSKkspNVkp1VwpFaGUWmjfvkYpFVjmOKWUekopFWL/eEqVSU1USm1RSg22TxIepJTa4on7qUypPZ1cD2+Y052jorCI8MG6VE83pVFatvUoPdsF0b1tRflPmjvExUFqqlFa79prYevWWq6y60KmDVBNRWpmPr3aBenhDZPq0KoZ1/Vrzycb0zlTWOLp5jQqh7MK2Jyew/VNYO2xhuK224yKErVaI8qFdIDyoNxzJew4eoYrerfVwxsmNmNMNPnFVj7ZWEnKk1Yny7dlADCxnw5QZjFpEgQGVjNZ1410gPKgTanZ2BR6WM/k+oa3ZER0CB+sS6XE6pyxj/i9J5kd37Tr/321LYPBkcGVFjDW3C8gAG680SgUce6cp1ujA5RHJaacxtfLwsAI3XMyuxljojl2xlirqL7eSTjAXR9s5B/f7+PWOetJTm1661HtPZ7Lr8dzub6/7j2ZzW23wZkz8M03nm6JDlAelZhymgGdWtHM18vTTdGqEdujDdFhzXl/TQr1KQ924GQe/+/H/ee/L7YqHli4mfhfT5KcmlXlopONyfJtR/GyCNde1r76gzW3Gj8e2rWD+fM93RIdoDzG8fxpRHSIp5ui1YDFItz9m87sPHqWxDouEX/2XAkzP95EM18Lft72xSG9BKtS3PXhRm56b32li042Jkoplm/LYFSXUMKC/DzdHK0cLy+jcOw330CWhzv3OkB5iH7+1PBMGdSRkOa+/POHvbXu6dhsisc+2Up6VgFzpg9h4QxjccjFM0ey7unLuapPW5S6sOhkY54Pt+VwDoezCpk0INzTTdEqcdttUFJiPIvyJB2gPMTx/GmQztxrMPx9vLi8Vxs2pWXz+vd7uXXOetYdMFYvTk7LrjJovfHTflbuOcmz1/ViuL02oiNz09fbwsyxXfD3ubDo5L4TufUaSjSz5Vsz8PW2cFWftp5uilaJAQOgQwd4+GGwWGqwDIeLmLVYbKOXmHKaARGt8PfRz58akrZBRp1EhfH86LZ5G+jWJpCUU/nYlMLHy8LCGRfPZ/th13He+Gk/UwZ15I5RURVe11Hcd/3BTHYcPcOyrRn4ell49cbLGtVCk6VWG19vP8blPdsQ5O/j6eZolVi4EE6dMnpRYFQ8nznT+NqdVSZ0gPIAx/OnB8d3q/5gzVRie7Zh7toUSkpteFksTOzfnvUppym1Gb2dolIbMz/exIRebRkY0Yr84lJe+3YvXcKa88oNfatcdsBRVV0pxb9X7ueNn/aTXVDCW9MGNpo3Mv9bn0ZmXhF9O5RfZEAzk2eeuRCcHAoKjO06QDVyG1Oz7M+fdIJEQ1PRMibJadnEvZ9IsdWGRYTI0AC+332cTzYdPn/ekexCdmWcrdFkbBHh0Su6Exroy/PLd3HD7HVc0bst43q0adCTuZPTsnn5G2Nts/+sOsCILq0b9P00ZlUtw7FggRGo0tONquevvOK6oKUDlAckpmQZz5/0/KcGaXC56uaDI4NZMOPioKWU4pVv9jBv7SEUFwoB1+YP8u0jo8gpKOFfP+5jz/Fc5qxJadDlsBJTTmO1P1crqcO/h+Y+EREVLWQIvr4wYwYUFhrfu3ror/EMbjcg+vlT41N+7S0R4ZrL2uPnY6lXIWAvi+AYFGzo2X092xkFYQVdGNnsKlqGw9sbioouBCcHx9CfK+gA5WZnz5Ww8+gZ/cvZBDiGA+tTCHhEdCh+ZbL7QgN9ndxK93E8p5s2PKJB9wSbgvLLcERGwocfXliSo7zKhgTrSwcoN9uknz81KfVd1dgR5P44viutA315J+EgeUWlTm6leySnZePrZeG53/bWwakBKLsMR2qq8X2NV+B1Eh2g3Ew/f9Jqa3BkMI9d2YO34wZzOKuAF5fv8nST6iQ5LZvLOrbUQ9sNWEVDfwEBrluBVwcoN9PPn7S6GtY5hAdiu/Jp8hG+2V7/orXuVFRqZceRM7rn1MBVtALvlCmuy+LTAcqN9PMnrb7+eHk3+ndqxZ8/305GTmH1J5jEzqNnKLba9MhBI+AY+rNa4cor4csvISPDNa9lugAlIiEi8oWI5ItImohMq+LYJ0Vkp4jkisghEXmy3P5UESkUkTz7xw+uv4PK6edPWn35eFl443cDKLUpZny8ka8OFjeIwrKONuoeVOMhAm+/bUzofeQR17yG6QIUMBsoBtoCccA7ItKnkmMFuB0IBq4GHhSRW8odM1EpFWj/uNJVja6JZVszLkob1rS6iGrdnLtGR7ErI5fP95c0iOrnyWnZRIYG6OrljUyXLvDss0ZR2W+/df71TRWgRKQ5MAV4TimVp5RaCywHpld0vFJqllJqs1KqVCm1F1gGjHZfi2suOTWL5VszsNqMpRXM/gdFM7cA+xpiCigpNff8KKUUyWnZDNbDe43Sk08ahWUnTnR+YVmzVZLoDpQqpfaV2bYNGFfdiWIUORsDvFdu1wIRsQBbgCeVUtsqOX8mMBMgLCyMhISE2re+CnO3n8NRm7q4xMailRvJ7eK5OS15eXlOv0ezacz36JdjxUvAqsAi4JeTRkLCEU83q0InC2xk5hUTVHSq1j+PxvwzdGjo97hyZRtOneqB1Wq8aUpLg7vvtrJnz14mTDhZr2ubLUAFAmfLbTsDBNXg3BcweoQflNkWB2zGGAp8GPheRHoqpXLKn6yUmgPMAejRo4eKiYmpbdur9Nq21UDu+aoCt04Y6tHx+ISEBJx9j2bTmO8xBujY7Rh/mL+ZKYM7cc8N/TzdpEp9vvkIsI1brxxOz3a1KxLbmH+GDg39Hu+889LCskVFXsyf35uXX+5dr2u7NUCJSAKV94bWAQ8B5f8HtwByq7nugxjPosYopYoc25VS68oc9qqI3IHRy/qqdi2vn+S0bPYcy+Wu0VG0DvQ7X69N0+rj6r7t6RliITHlNEqpKiule9KmtGyC/Lzp1qYm7zO1hqaqwrL15dYApZSKqWq//RmUt4h0U0rtt2/uD1Q6M1FEfg/8CRirlKpujEOB+3MU3v35IK0CfHjiyh409zNbp1VryEZ38GbezgI2p+eY9k3P5rRsBkYG42UxZwDV6qeywrLOqC5hqiQJpVQ+8Dnwkog0F5HRwCTgfxUdLyJxwN+AK5RSKeX2RYjIaBHxFRF/ewp6a4yemtvsP5HLj7tPcMfIKB2cNKcb0s4bfx+LfRjNfM4UlrD3RK5OkGjEXFldwlQByu5+oBlwElgE3KeU2gUgImNEJK/MsS8DocDGMnOd3rXvCwLeAbKBoxhp6Ncopdya7vTuzyn4+1gqXUlV0+qjmbdwVZ92fLUtg6JSq6ebc4mth3NQSs9/aswqKiw7Z45zqkuY7i29UioLmFzJvjUYiRSO7ztXcZ1dgEefHGfkFLJs61FuGxFJSPOGW4VaM7cbB3Vk2dYMVu05yTWXtfd0cy6SnJaNRWBARCtPN0Vzobg4vR5UgzN3zSEA7hlTaRzVtHr7TdfWtAnyY+nmo55uyiWS07Lo2a4FgXp4W6sDHaBcJDu/mEVJ6Vw/oAMdgwOqP0HT6sjLItwwMJyEvSc5nVdU/QluUmq1sdXEyRua+ekA5SIfr0+jsMTKH8Z18XRTtCbgxkEdKbUpvtrmoqqddbD3RC75xVaGROkApdWNDlAu8MuBTN75+QBDIoPp3lbP/dBcr0e7IPp0aMHnW8wzzOco56UrmGt1pQOUkyWnZXP7f5M4V2Jj+9Ezuuae5jY3DurI9iNn2H+iynntbpOclk2bID86BjfzdFO0BkoHKCdLTDlNqc2oume1mruIp9a4XN+/AxaBZ7/caYo3Rslp2QyJCjZthQvN/HSAcrLIUCMhQjBq7unFCTV3Sc8qAGDDoSyPL8GxcvcJjmQX0kYvr6HVgw5QTnbgpDGP+A/jurDgnhE6g0lzG6Mmn/F1sQeX4EhOy+a+BckALEw6bIrenNYw6QDlREoplm3NYGR0KE9f01MHJ82tRkSH4udjueh7T1h/MJMSqx7m1upPBygn2nH0DIcy85k8sIOnm6I1QYMjg1lwzwiGRgUjQESIZ+bfeXsZf1ZE9DC3Vj86QDnRl1sy8PWycHVfc5Wb0ZqOwZHBvHrjZVgVLPVAAVmlFN/uOEabIF8eu6K7HubW6kUHKCex2hRfbc8gtmcYLZv5eLo5WhPWtU0QQ6OC+WTjYZTjoZSbrNmfybYjZ3j0ih48NL6bDk5avegA5SS/HMzkVG4RkweEe7opmsYtQyM4lJnPhkNZbn3dt1YdoH1Lf24cpH8PtPrTAcpJvtySQZC/N7E923i6KZrGtZe1J8jfm082HnbbayamnCYpNYt7x0bj5+3lttfVGi8doJzgXImV73cd55q+7fD30b+Ymuc18/Vi8oBwVuw4xpmCEre85lurDtA60JdbhjlhKVVNQwcop1i55wR5RaV6eE8zld8N7URRqY0vtrg+WWJzejZrD2QyY0y0fpOmOY0OUE6wbGsGbVv4MVyn02om0je8JZeFt2SxG5IlZq86QKsAH+JGRLr0dbSmxXQBSkRCROQLEckXkTQRmVbFsS+ISEmZ5d7zRCS6zP4BIpIsIgX2zwOc3d6cgmIS9p7k+v4d8LLommOaudwyrBO/Hs9l25EzLnuNTzcd5qdfT3JNn3Z6YULNqUwXoIDZQDHQFogD3hGRPlUc/4lSKrDMRwqAiPgCy4D5QDDwEbDMvt1pVuw4TolVMUkP72kmdH3/DjTz8eKTjekuuX5yWjZPL90OwOdbjuqyRppTmSpAiUhzYArwnFIqTym1FlgOTK/D5WIAb+DfSqkipdSbGDVcxzurvQD/S0wlpLkPRSVWZ15W05wiyN+H3/Zrz/KtGeQXlTr9+j/tOYG9eD+luqyR5mRm6493B0qVUvvKbNsGjKvinIkikgUcA95SSr1j394H2K4uHnzfbt/+XfmLiMhMYCZAWFgYCQkJ1TZ24/ES9hwrBuDWOet5aqg/XYMbxgPivLy8Gt1jQ6bv0dDD28qnxVZuf3sl10T5OPX/6NHDxv9/AbwE/HLSSEhwXlKG/hk2bWYLUIHA2XLbzgCVLUu7BJgDnACGA0tFJEcptch+rfID75VeSyk1x34tevTooWJiYqpt7OtvrsEYjQSrgqJWkcTEdK32PDNISEigJvfYkOl7NASlZiEb1pN8wsquLOXU8kPLTmwlyP8E946NZmSX1k6vHKF/hk2bW4f4RCRBRFQlH2uBPKBFudNaABUuEaqU2q2UylBKWZVSvwBvADfZd9fqWrV1KDOf3cfO4mURvHRRTM3EEstUk3DmMhw2m2LN/kxie7ThQV3WSHMBt/aglFIxVe23P4PyFpFuSqn99s39gV01fQmM0Qbs5zwuIlJmmK8fRhJGvf3np/34elt4a9og9h7PZUR0qP4F1UxpRHQovt4WikptiIjT3kjtOX6WzLwixnYPc8r1HGw2G5mZmeTk5NCyZUv27Nnj1OubTUO/Ry8vL1q1akXr1q2xWJzb5zHVEJ9SKl9EPgdeEpF7gAHAJGBURceLyCRgNZADDAX+CPzFvjsBsAJ/FJF3gRn27avq284DJ/P4cutR7hkTzYRebZnQq219L6lpLjM4MpiFM0bw1GfbOZ1fRP+OLZ1y3TX7MwEY0621U67ncOTIEUSEqKgozp07R4sW5QdCGpfc3FyCgip7imFuSilKSko4ceIER44cISLCuVVETJXFZ3c/0Aw4CSwC7lNK7QIQkTEiklfm2FuAAxjDdh8DrymlPgJQShUDk4HbMQLY74HJ9u318sZP+/H38eLesdHVH6xpJjA4Mpinru5BTkEJP+875ZRrrt53ip7tgmjbwt8p13PIz88nPDwcX19fRPTcQjMTEXx9fQkPDyc/P9/p1zdVDwpAKZWFEVgq2rcGI/nB8f2t1VxrCzDYme3bezyXr7dncN+4LoQG+jnz0prmUuN7tqF1oC+fbjrC5fXs9RcUl7IpNZs7RrmmcoSzh4o013LVz0v/L6ilN37aR3Nfb2aM0b0nrWHx8bIweUA4P/16gqz8+g0kbEjJothqc/rzJ00rSweoWtidcZYVO47z+9FRBDd3akEKTXOLm4d0osSq+HLL0XpdZ/X+U/h5WxgaFeKklmnapXSAqoV/r9xHkL83d+vek9ZA9WgXRL+OLVmyqX4FZFfvO8Xw6FBdubyBeOGFF+jbt6+nm1FrOkDV0KcbD/PD7hNcd1l7vaS71qDdPLgjvx7PZVdG+TnxNXM0p5CDp/IZ6+TsvcZi8+bNeHl5MXr06FqdFxMTw4MPPuiiVjVMOkDVQHJaNk9/bhTE/FIXxNQauOv7h+PrbeHTTXVbbXeNPQvQ7M+fFiyAqCiwWIzPCxa453Xnzp3L/fffz86dOxv0/CYz0AGqBr7alnG+IGaJLoipNXAtA3y4sndblm3LoKi09kWO1+zPpF0Lf7q1Caz+YA9ZsABmzoS0NFDK+DxzpuuDVGFhIQsXLmTmzJncdNNNzJs376L9iYmJjB8/nubNm9OyZUvGjx/PsWPHuPPOO/n555+ZPXs2IoKIkJqaSkJCAiJCZmbm+WukpqYiImzatAkAq9XK3XffTefOnWnWrBndunVj1qxZ2Gw2196sG5guzdyMUk4ZU690SSOtsbh5SCe+3n6MlbtPcl2/9jU+z2pTrD2QyZW927p1jtIjj8DWrTU/PjERioou3lZQAHffDe+/X7NrDBgA//53zV8T4LPPPiMyMpLLLruM6dOnM3XqVF599VV8fHzYtm0bsbGxTJ8+nX/961/4+fmxevVqSktLeeONN9i3bx89e/bkb3/7G2AUrU5NTa32NW02G+Hh4SxZsoSwsDCSkpKYOXMmoaGh3H333bW7AZPRAaoah7MKWHfwNBP7tadn+xa6pJHWKPyma2vat/Tn0+TDtQpQ24/kcKawxPTDe+WDU3XbnWXevHlMn26sDjRu3DgCAgJYtmwZN910E7NmzWLAgAHMmTPn/PG9evU6X0nC19eXgIAA2rVrV6vX9PHx4aWXXjr/fVRUFJs3b2bRokU6QDV289YewiLwl+t60b5lM083R9Ocwssi3DgonLfjD/Lat78yoXfbGr3xWr0vExEjwLlTbXsyUVHGsF55kZHgqpUtDhw41ZZQmwAAE6NJREFUwNq1a1m4cCFgVFmIi4tj3rx53HTTTWzZsoUbbrjBJa/97rvvMnfuXNLS0igsLKSkpITISNdMonYnHaCqkJVfzOKN6UwaEK6Dk9bo9G7fAgW88/NBPvjlUI2W4Viz/xT9wluafh7gK68Yz5wKCi5sCwgwtrvK3LlzsVqtF9Wjc6TyHz5ct4QUR4WGslMCSkpKLjrmk08+4ZFHHuH1119n1KhRtGjRgtmzZ/PFF1/U6TXNRAeoKnz4SyrnSmz8YZye96Q1PqmnL/z1dizDUVWAOnuuhC2Hc7hvXBd3NK9e4uKMz888A+npEBFhBCfHdmcrLS3lo48+4tVXX+W3v/3tRfumT5/OBx98wMCBA1m1qvJa1b6+vlitFyethIUZQ6nHjh07//XWcg/j1q5dy/Dhwy9KUT948GC97scsdICqREFxKR+vT2VCr7Z0bdMwKw1rWlVGRIfi723hXKkNpWBQRKsqj5+/PhWrTdHOycVhXSUuznUBqbxvvvmGzMxMZsyYQWjoxUlUt9xyC++++y6ff/45I0eOZObMmTzwwAP4+/uzZs0aRo0aRe/evYmKiiIpKYnU1FQCAwMJCQmha9eudOrUiRdeeIG///3vpKam8vLLL190/e7du/Phhx/y7bff0rVrVxYvXszPP/9McHDDf1au08wrsTjpMDkFJdwXo3tPWuM0ODKYBTNGMHVIRxTw877MSo9NTs3i9R/2AfDyit16LmA58+bNIzY29pLgBHDzzTeTmprKqVOnWLlyJb/++isjRoxg+PDhLF68GB8fY+L/E088ga+vL7179yYsLIz09HR8fHxYvHgxKSkp9O/fn+eff/58lp/Dvffey9SpU5k2bRpDhw4lNTWVxx9/3C337WpSn3InjVX3Hj1U2F3vEN6qGUv+MNLTzXGJprDMtL7HmvvT0u0s2XSYL+4fTf9OF/ekrDbFze/+wub0HMCYbvHYlT14ILZrvV+3Inv27KFXr15Aw14rqaYayz2W/bmVJyLJSqkhtb2m7kFVIL9EcTSnkD/o3pPWRPzlul60CfLnyc+2XTR591yJlYcWbWZzeg5eFtFzATW30s+gKpB9TjEkOIDYHm083RRNc4sW/j68ckNf7v5oE7PjD/LYFd05U1jCjI83kXQoi2ev68XAiGASU07ruYCa2+gAVQGrgmNnC9mcnqN/EbUm4/JebblhYDizV+3naHYBGw9lc+xsIW/eOpDr+3cA0L8PmlvpIb5K2GxK19zTmpxJAzpgVbB081HSswt45tpe54OTprmb6QKUiISIyBciki8iaSIyrYpjvxWRvDIfxSKyo8z+VBEpLLP/h//f3r0HR1XlCRz//prEDgQi8kgzyTiEDI8JII8woi4PYcUSJjrsLr7QZWUV44ZHiZbjUIUCPgrU4uXOiivISIQpZimGZVdd3CoRFmHF4iGuG6MosHHNY1ddiIEAceC3f9ybptMmPDrd6duX36fqVvqec+/N+aW78+s+99xzL7Yd1s9uLkdlVd/ROMNeQOBEw6VPJmtMvHixi+8loAEIAUOAt0TkI1Uti95QVSdErovIdiD6SrjbVPWdS2nAVUG5qKvqjfGb6/O7EkwP8P0fz9qHNJN0nkpQIpIJTAIGqupxYKeI/DMwBZhzgX3zgFHA1Na248qgWHIyl6VhPa/id9Out8EQxhM8dR2UiAwFdqlqh4iyx4AbVfW2C+w7D/hTVR0TUfZfQHucrswPgV+p6kct7F8MFAN079592IYNG1oXjMcdP36cjh29ez+feLAYU9OVV15J797ONVZnzpyhXTt/31beLzF+8cUX1NbWNls3duzYmK6D8tQ3KKAjEH0f6lrgYq5i+yvg2aiye4H9gAAPA/8qIj9T1WPRO6vqSmAlQL9+/dQu8Ex9FmNqKi8vD1+46peLWM/HLzFmZGQwdOjQuB6zTQdJiMh2EdEWlp3AcSArarcsoO4Cxx0J9AA2Rpar6i5VPamq9aq6CDiG0w1ojPGjF16AbdvOv822bc52xvPaNEGp6hhVlRaWkcBBIE1E+kTsNhj4wQCJKPcBm9zzVudtAtB2twE1xrSta6+FO+9sOUlt2+bUX3ttQn791KlTw7dsj1yiZyBPJSLCxo0bL7xhAnhqmLmqngA2AU+LSKaIjAAmAmtb2kdE2gN3Amuiyn8iIiNE5AoRyRCRXwHdgF0JC8AYk1xjx8KGDc0nqcbktGGDs12CjBs3jurq6ibLwIEDL/k4DQ0NCWhdavFUgnJNxxnY8L/AeqCkcYi5iIwSkehvSX+G03UX/ZGpE/AycBSoBMYDE1TVrr41xs+aS1JtlJwAgsEgPXr0aLKkpaWxY8cOrrvuOjIyMgiFQjzyyCNNktCYMWMoKSnhscceo3v37owYMQKATz75hKKiIjp16kR2djaTJ0+mpqamye8sLS3lmmuuIRgMEgqFuO+++8J1S5cuZdCgQWRmZpKbm8u0adM4duzcafja2lqmTJlCdnY2GRkZ5Ofns9y9hXFeXh7gzMguIuH1tuK1QRKo6v/hJJ3m6t7DGUgRWbYeJ5FFb1sGDEpEG40xbWz2bLjUbrKcHLjlFvjRj6C6GgoK4KmnnOViDBly6feab0FlZSUTJkxgypQprFmzhkOHDjFt2jQCgQALFiwIb7du3TqKi4t57733UFWqq6sZPXo0DzzwAIsXL+b7779n7ty5TJw4kffff59AIMArr7zCww8/zMKFCykqKuL48eNNbowYCARYvnw5+fn5VFRUMGvWLGbNmsXatU7H1BNPPMHHH3/Mm2++SSgU4siRI3z99dcA7Nmzh+zsbFatWsWtt97a5qMNPZegjDEmLq66yklOjbfUbaMb+L399ttNhv6PGjWKwsJCcnJyWLFiBYFAgIKCAp577jkeeughHn/88fAovl69erFkyZLwvvPmzWPw4ME8//zz4bLXX3+dLl26sHfvXoYPH84zzzzD7NmzefTRR8PbDBs2LPx49uzZ4cd5eXm88MILTJw4kdLSUgKBABUVFRQWFjJ8+HAAevbsGd6+8S6+nTt3pkePHvH6E100S1DGGO+L5ZtMY7fek0/Cyy/D/PkJ794DGD16NCtXrgyvt2/fnlmzZnH99dcTCJw7qzJy5EgaGho4fPgwoVAIaJpYAPbt28eOHTuavdbt0KFD5OXlUVlZyU033dRie959910WLVpEeXk5tbW1nDlzhoaGBmpqasjJyaGkpITbb7+dffv2cfPNN3Pbbbdx4403tvbPEBeWoIwx/hN9zmns2DY7B9WhQ4fwhcYXQ+TcwOLMzMwmdWfPnqWoqIjFixf/YL9QKMTJkyfPe+yKigqKiop48MEHefrpp+natSv79+9n8uTJ4fNfEyZMoKKigi1btrB161aKioq44447eO211y46hkTx4iAJY4yJXXMDIs43uq8NFBQUsHv3bs6ePRsu27lzJ1dccQW9evVqcb/CwkLKysro2bMnvXv3brI0DprIzc1l69atze6/d+9eGhoaWLZsGTfccAN9+/alqqrqB9t169YtfH5s9erVlJaWcvr0aQDS09M5cyY5kwZbgjLG+Mf5RuslMUlNnz6dqqoqpk+fTnl5OW+99RZz5sxh5syZdOjQocX9ZsyYQW1tLXfddRcffPABhw8f5p133qG4uJi6Omf+grlz57J8+XKWLVvGwYMHOXDgQPg8Vp8+fTh79izLly/nyJEjrF+/PjxCr9G8efPYvHkzn3/+OeXl5WzatIn8/HyCwSDgnLfaunUrNTU1HD16NEF/oeZZgjLG+MeePefvxmtMUnv2tGmzcnNz2bJlCx9++CFDhgzh/vvvZ/LkySxcuPC8++Xk5LBr1y4CgQDjx49nwIABzJgxg2AwGE4gJSUlvPTSS6xatYqBAwcyfvx4ysqcuQ0GDRrEiy++yNKlS+nfvz+vvvrqD7oLg8Egc+fOZfDgwYwYMYK6ujreeOONcP2SJUvYtm0bV199ddynMroQT00W6xX9+vXTzz77LNnNSCg/zuEWzWJMTeXl5RQUFAD+mafufPwSY+TzFk1EYpos1r5BGWOM8SRLUMYYYzzJEpQxxhhPsgRljDHGkyxBGWM8xwZvpZZEPV+WoIwxnpKenn7BGRKMt5w8eZL09PS4H9cSlDHGU7Kzs6msrKS+vt6+SXmcqlJfX09lZSXZ2dlxP77NxWeM8ZSsrCwAqqqqqKurIyMjI8ktSqxTp06ldIzp6emEQqHw8xZPlqCMMZ6TlZVFVlYW27dvb/PZC9ra5RBjrKyLzxhjjCdZgjLGGONJnktQIjJTRPaKyGkRWXMR2z8iIjUi8p2I/FZEghF1eSKyTUTqReRTERmX0MYbY4yJG88lKKAKeBb47YU2FJFbgDnATUBPIB94KmKT9cCHQFdgLrBRRLrHu8HGGGPiz3MJSlU3qepm4NuL2Pw+YLWqlqnqUeAZYCqAiPQFCoH5qnpSVf8AfAxMSkzLjTHGxFOqj+IbAPxTxPpHQEhEurp1h1W1Lqp+QHMHEpFioNhdPS0i/5mA9npJN+CbZDciwSzG1Of3+ODyiLFfLDuleoLqCNRGrDc+7tRMXWN9bnMHUtWVwEoAEdkby71LUonF6A9+j9Hv8cHlE2Ms+7VpF5+IbBcRbWHZGcMhjwORV4c1Pq5rpq6xvg5jjDGe16YJSlXHqKq0sIyM4ZBlwOCI9cHA/6jqt25dvoh0iqoviz0CY4wxbcVzgyREJE1EMoB2QDsRyRCRlroiXwceEJH+ItIZeAJYA6CqB4EDwHz3GH8ODAL+cBHNWNnaOFKAxegPfo/R7/GBxdgi8dpkjCKyAJgfVfyUqi4QkZ8AnwD9VfVLd/tHgV8D7XGSz9+o6mm3Lg8nYV0HfAnMUNV3Eh+FMcaY1vJcgjLGGGPAg118xhhjDFiCMsYY41GWoCKISBcR+UcROSEiFSJyT7Lb1Frnm9tQRG5y5yisd+cs7JmkZsZMRIIistp9vupE5ICITIioT/kYAURknYhUu3NOHhSRaRF1vogRQET6iMgpEVkXUXaP+/yeEJHNItIlmW1sDfdSm1MictxdPouo80WcInK3iJS7cRwSkVFu+SW/Ti1BNfUS0ACEgHuBl0Wk2ZknUkizcxuKSDdgE/Ak0AXYC/xDm7eu9dKA/wZuBK7EGcm5wZ0o2C8xAiwC8lQ1C/gl8KyIDPNZjOC8B/c0rrjvv1eAKTjvy3pgRXKaFjczVbWju/QD/8QpIjcDzwN/jTNhwmjgcKyvUxsk4RKRTOAoMNAdoo6IrAUqVXVOUhsXByLyLPBjVZ3qrhcDU1X1T9z1TJzpVoaq6qdJa2gciMh/4Ewa3BUfxigi/YDtwMNAZ3wSo4jcDfwFzkjd3qr6lyKyECcx3+Nu81OgHOgaNY1ZShCR7cA6VX01qtwXcYrIv+PMj7o6qjym/zf2DeqcvsAfG5OTq8W5+3xgAE58AKjqCeAQKR6viIRwnssyfBajiKwQkXrgU6Aa+Bd8EqOIZAFPA49GVUXHdwinl6Nv27Uu7haJyDcisktExrhlKR+niLQDfg50F5EvROQrEfk7EWlPjK9TS1DndAS+iyqrxfma6kctzVWYsvGKSDrwO6DU/VTmqxhVdTpO20fhdJecxj8xPoPzyfurqHK/xNfo1zi3BcrFuXj1Dffbkh/iDAHpwO04r9EhwFCcbveY4rMEdc7lNnefr+IVkQCwFudT50y32FcxAqjqGVXdCfwYKMEHMYrIEGAcsKyZ6pSPL5KqfqCqdap6WlVLgV3AL/BHnCfdn79R1WpV/QZYSiviswR1zkEgTUT6RJT5ee6+JvMYun3CPyUF4xURAVbjfIKbpKrfu1W+ibEZaZyLJdVjHAPkAV+KSA3wGDBJRPbzw/jygSDO+9UPFBB8EKd7T76vcGIKF7s/Y3udqqot7gL8HucuvJnACJyvoAOS3a5WxpQGZOCMAlvrPk4DurvxTXLLngd2J7u9Mcb498BuoGNUuS9iBLKBu3G6SdoBtwAncEbzpXyMQAegR8SyGNjoxjYAp+t9lPu+XAf8PtltjjHOzu5z1/gevNd9Hvv6JU6c84h73NfsVcB7ON23Mb1Okx6Qlxac4Y+b3RfNl8A9yW5THGJagPMpJnJZ4NaNwznhfhJnVFhestsbQ3w93ZhO4XQjNC73+ijG7sC/Acfcf2IfAw9G1Kd8jFHxLsAZ6da4fo/7fjyBc4PSLsluYyuexz043VrHcD5U3eynOHHOQa1w46sB/hbIcOsu+XVqw8yNMcZ4kp2DMsYY40mWoIwxxniSJShjjDGeZAnKGGOMJ1mCMsYY40mWoIwxxniSJShjjDGeZAnKmBQkIlkiskBECpLdFmMSxRKUManp58B8nCv3jfElS1DGpKahOLfb+CTZDTEmUWyqI2NSjIiUAz+LKt6kqpOS0R5jEsUSlDEpRkSuxZl5vwxY6BZXq2pF8lplTPylJbsBxphL9hHODQt/o6q7k90YYxLFzkEZk3oGAFcA+5PdEGMSyRKUMamnEOceWAeS3RBjEskSlDGpZyhwSFW/S3ZDjEkkS1DGpJ7+2PBycxmwQRLGpJ5jQKGI3ALUAp+r6rdJbpMxcWfDzI1JMSIyEFgNDAIygFGqujO5rTIm/ixBGWOM8SQ7B2WMMcaTLEEZY4zxJEtQxhhjPMkSlDHGGE+yBGWMMcaTLEEZY4zxJEtQxhhjPMkSlDHGGE/6f+b8rlFeX9iEAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "def plot_multiple_forecasts(X, Y, Y_pred):\n", " n_steps = X.shape[1]\n", " ahead = Y.shape[1]\n", " plot_series(X[0, :, 0])\n", " plt.plot(np.arange(n_steps, n_steps + ahead), Y[0, :, 0], \"bo-\", label=\"Actual\")\n", " plt.plot(np.arange(n_steps, n_steps + ahead), Y_pred[0, :, 0], \"rx-\", label=\"Forecast\", markersize=10)\n", " plt.axis([0, n_steps + ahead, -1, 1])\n", " plt.legend(fontsize=14)\n", "\n", "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n", "save_fig(\"forecast_ahead_plot\")\n", "plt.show()" ] }, { "cell_type": "markdown", "source": [ "As you might expect, the prediction for the next step will usually be more accurate than the predictions for later time steps, since the errors might accumulate. If you only want to forecast a few time steps ahead, on more\n", "complex tasks, this approach may work well." ], "metadata": { "id": "SkRQp2eMLsDS" } }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "r3mvSba_5jk-" }, "outputs": [], "source": [ "np.random.seed(42)\n", "\n", "n_steps = 50\n", "series = generate_time_series(10000, n_steps + 10)\n", "X_train, Y_train = series[:7000, :n_steps], series[:7000, -10:, 0]\n", "X_valid, Y_valid = series[7000:9000, :n_steps], series[7000:9000, -10:, 0]\n", "X_test, Y_test = series[9000:, :n_steps], series[9000:, -10:, 0]" ] }, { "cell_type": "markdown", "metadata": { "id": "HDbQm3hU5jk-" }, "source": [ "Now let's predict the next 10 values one by one:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "id": "Nb9XFiww5jk_" }, "outputs": [], "source": [ "X = X_valid\n", "for step_ahead in range(10):\n", " y_pred_one = model.predict(X)[:, np.newaxis, :]\n", " X = np.concatenate([X, y_pred_one], axis=1)\n", "\n", "Y_pred = X[:, n_steps:, 0]" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "id": "ioAnpOO75jk_", "outputId": "e2c25792-d471-4410-e9c7-639ea3b9bc71", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(2000, 10)" ] }, "metadata": {}, "execution_count": 30 } ], "source": [ "Y_pred.shape" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "id": "JSeX3I_y5jk_", "outputId": "9c0fe992-efbc-4dd4-cf9b-c6e34d711664", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.19622587" ] }, "metadata": {}, "execution_count": 31 } ], "source": [ "np.mean(keras.metrics.mean_squared_error(Y_valid, Y_pred))" ] }, { "cell_type": "markdown", "metadata": { "id": "mLeMjfsH5jk_" }, "source": [ "Let's compare this performance with some baselines: naive predictions:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "id": "oYYPT19r5jk_", "outputId": "3c3443e1-ff3a-4da8-da44-54a5c20273c1", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0.25697407" ] }, "metadata": {}, "execution_count": 32 } ], "source": [ "Y_naive_pred = np.tile(X_valid[:, -1], 10) # take the last time step value, and repeat it 10 times\n", "np.mean(keras.metrics.mean_squared_error(Y_valid, Y_naive_pred))" ] }, { "cell_type": "markdown", "metadata": { "id": "Hb7kFoGB5jk_" }, "source": [ "The second option is to train an RNN to predict all 10 next values at once. We can still use a sequence-to-vector model, but it will output 10 values instead of 1. Now we just need the output layer to have 10 units instead of 1:" ] }, { "cell_type": "code", "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n", " keras.layers.SimpleRNN(20),\n", " keras.layers.Dense(10)\n", "])\n", "\n", "model.summary()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3JAX6ywcMpTZ", "outputId": "094bf7d4-fef8-4c70-faec-59eadcab1aba" }, "execution_count": 34, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_4\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " simple_rnn_6 (SimpleRNN) (None, None, 20) 440 \n", " \n", " simple_rnn_7 (SimpleRNN) (None, 20) 820 \n", " \n", " dense_2 (Dense) (None, 10) 210 \n", " \n", "=================================================================\n", "Total params: 1,470\n", "Trainable params: 1,470\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "id": "v6AHQGbF5jk_", "outputId": "8bacd10d-4c87-41c4-a0b1-42ac4eb4c5fa", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 34s 149ms/step - loss: 0.0669 - val_loss: 0.0317\n", "Epoch 2/20\n", "219/219 [==============================] - 36s 164ms/step - loss: 0.0265 - val_loss: 0.0200\n", "Epoch 3/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0183 - val_loss: 0.0160\n", "Epoch 4/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0155 - val_loss: 0.0144\n", "Epoch 5/20\n", "219/219 [==============================] - 25s 114ms/step - loss: 0.0139 - val_loss: 0.0118\n", "Epoch 6/20\n", "219/219 [==============================] - 36s 167ms/step - loss: 0.0128 - val_loss: 0.0112\n", "Epoch 7/20\n", "219/219 [==============================] - 25s 115ms/step - loss: 0.0122 - val_loss: 0.0110\n", "Epoch 8/20\n", "219/219 [==============================] - 25s 114ms/step - loss: 0.0115 - val_loss: 0.0103\n", "Epoch 9/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0111 - val_loss: 0.0112\n", "Epoch 10/20\n", "219/219 [==============================] - 24s 111ms/step - loss: 0.0110 - val_loss: 0.0100\n", "Epoch 11/20\n", "219/219 [==============================] - 25s 116ms/step - loss: 0.0108 - val_loss: 0.0103\n", "Epoch 12/20\n", "219/219 [==============================] - 26s 120ms/step - loss: 0.0102 - val_loss: 0.0096\n", "Epoch 13/20\n", "219/219 [==============================] - 43s 195ms/step - loss: 0.0104 - val_loss: 0.0100\n", "Epoch 14/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0098 - val_loss: 0.0103\n", "Epoch 15/20\n", "219/219 [==============================] - 35s 161ms/step - loss: 0.0095 - val_loss: 0.0107\n", "Epoch 16/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0092 - val_loss: 0.0089\n", "Epoch 17/20\n", "219/219 [==============================] - 25s 113ms/step - loss: 0.0094 - val_loss: 0.0111\n", "Epoch 18/20\n", "219/219 [==============================] - 25s 113ms/step - loss: 0.0095 - val_loss: 0.0094\n", "Epoch 19/20\n", "219/219 [==============================] - 25s 116ms/step - loss: 0.0093 - val_loss: 0.0083\n", "Epoch 20/20\n", "219/219 [==============================] - 25s 113ms/step - loss: 0.0094 - val_loss: 0.0085\n" ] } ], "source": [ "model.compile(loss=\"mse\", optimizer=\"adam\")\n", "history = model.fit(X_train, Y_train, epochs=20,\n", " validation_data=(X_valid, Y_valid))" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "id": "CTOKPakr5jlA" }, "outputs": [], "source": [ "np.random.seed(43)\n", "\n", "series = generate_time_series(1, 50 + 10)\n", "X_new, Y_new = series[:, :50, :], series[:, -10:, :]\n", "Y_pred = model.predict(X_new)[..., np.newaxis]" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "id": "J2sKchZu5jlA", "outputId": "8ac7905d-f1cd-4f9b-e2e4-742405e7668a", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEUCAYAAABpmDIHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxU1fn48c8z2UkIJCHskLAjKCC7IppY61JBXDAqiLu0Wmtt1da6fEv9WanW+rK2FkWooiKIioJaNyRhNSxhlR2ysUM2IAkhyzy/P+4khJBAEmYyd4bzfr3mlcy52zlkyJN77jnPEVXFMAzDMLzF4e0KGIZhGOc3E4gMwzAMrzKByDAMw/AqE4gMwzAMrzKByDAMw/AqE4gMwzAMrzKByDAMw/AqWwYiEXlERFaLyAkRefcs+/5ORA6IyFER+a+IhFTbFi8iySJSLCJbReQqj1feMAzDaBBbBiJgH/AC8N8z7SQi1wBPAT8D4oCuwF+q7TILWAvEAM8An4hIrCcqbBiGYTSO2Dmzgoi8AHRU1Xvq2P4hkKmqT7ve/wyYqaptRaQnsBFoparHXNuXuLa/2SQNMAzDMM4q0NsVOEd9gXnV3q8H2ohIjGtbemUQqra9b20nEpGJwESA0NDQQZ07d/ZMjW3A6XTicNj1Zvjcmfb5Ln9uG/h/+7Zv356jqg3udfL1QBQBHKn2vvL75rVsq9zeobYTqepUYCpAr169dNu2be6tqY2kpKSQkJDg7Wp4jGmf7/LntoH/t09EshpznK+H5kIgstr7yu+P1bKtcvsxDMMwDNvw9UC0Cehf7X1/4KCq5rq2dRWR5jW2b2rC+hmGYRhnYctAJCKBIhIKBAABIhIqIrV1I74H3C8ifUSkJfAs8C6Aqm4H1gF/dh1/E9AP+LRJGmEYhmHUiy0DEVZAOY41NPtO1/fPikhnESkUkc4AqvoN8DKQDGQDWcCfq53ndmAwkA/8DRirqoebrBWGYRjGWdlysIKqTgIm1bE5osa+rwKv1nGeTCDBfTUzDMMw3M2ud0SGYRjGecIEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMoEIsMwDMOrTCAyDMMwvMq2gUhEokXkMxEpEpEsERlXx35fu5YPr3yVisjGatszReR4te3fNV0rDMMwjLOx5VLhLm8ApUAbYADwlYisV9VN1XdS1euqvxeRFGBhjXONVtUFHqyrYRiG0Ui2vCMSkXDgFuA5VS1U1aXAfGDCWY6LB0YC73m6joZhGIZ72DIQAT2BclXdXq1sPdD3LMfdBSxR1cwa5TNF5LCIfCci/d1YT8ONlu3M4ZVvt5GWle/tqhiG0YREVb1dh9OIyEjgY1VtW63sQWC8qiac4bidwAuq+m61shHAGkCA37pevVW1oMaxE4GJALGxsYPmzJnjtvbYTWFhIREREd6uximSs8uYsbkUgCAH/HFIKN2jAhp1Lju2z538uX3+3Dbw//YlJiamqerghh5n12dEhUBkjbJI4FhdB4jIZUBb4JPq5aq6rNrbySJyN1b33Rc19psKTAXo1auXJiQkNLbutpeSkoJd2qeqvLs8k/e3bK4qK3fCiZZxJCR0b9Q57dQ+T/Dn9vlz28D/29dYdu2a2w4EikiPamX9gU117A9wNzBXVQvPcm7FujsyvKzoRDm/mbWWv3yxmYGdowgJPPlxHBIf5cWaGYbRlGx5R6SqRSIyF3heRB7AGjU3Bri0tv1FJAxIAm6qUd4Z6ASswgq6vwFaActqnsNoWvPX7WXSF5vJLyrlj9f25peXd2Xt7gLe/zGTz9ftY/+REm9X0TCMJmLXOyKAh4Ew4BAwC3hIVTeJyEgRqXnXcyNQACTXKG8OTAHygb3AtcB1qprr0ZobZ/T52r08OnsdeUWlBAU4GNolGodDGBQXxatJA+jZJoI3knfidNrv+aVhGO5n20CkqnmqeqOqhqtqZ1X90FW+RFUjauw7S1XjtMbIC1XdpKr9XOeIUdWfqerqpmyHcbp//rCj6vsKp5PU9JN/FzgcwsMJ3dl+sJDvtxz0RvUMw2hitg1Ehn9avjOHjJwiAh1CgEBQoIPhXWNO2WdUv3Z0jm7GG8k7seOoTsMw3MsEIqPJOJ3K5K+30qFlGO/fP5TfX92LmQ8MZ1DcqQMTAgMcPJzQjQ17jrB4R46XamsYnjdzJsTHg8NhfZ05s/Yyf2fLwQqGf/py43427j3Cq0n9uaRbKy7p1qrOfW8e2JF//rCDNxbu5IqesU1YS8Nwv5kz4ZlnIDv7Cjp3hr/+1SqfOBGKi63vs7Lg3ntBBEpLT5ZNnGh9P35809e7qZg7IqNJnCiv4O/fbuWCdpHcOKDDWfcPDnQw8fKurMzMY0W6GVti+K6ZM61gkpUFqkJWFtx3HzzwwMkgVKms7GQQqlRcbAUxf2YCkdEkZqZmszvvOE9d1xuHo37TuG4f0pmY8GBe/N8W3kjeaVL/GD7pmWdODzilpVDSgBkK2dnurZPdmEBkeNzRkjL+tXAHl3VvxeU96u6OqyksOIDrLmzH+j1H+Md32xg/LdUEI8N3vPwyJCefFkSe5GUSXDNNEkjmSV4+7dCa5Z07e7SmXmcCkeFxb6bsIr+4jKeu641Iw5JaxEQEAeBUKC0/dai3YdjakCGQlMRtrU+d3riKIcwhiWeavcockljFkKptQUHw88DkU8oDAk4+U/JXJhAZHvX9poNMXZzOyB6tuLBDiwYff3nP1lWpf1ShX8eGn8MwvCIxEebM4Z3iJK4NORmMUkjklaA/8f+OP0H2uD+REZeICMTFwf+eTGZ+WBKPtpnDIkmkeXOoqICLL/ZiO5qACUSGx6Rl5fOrmWmUO5WVGXmN6lYbFBfFhw8O59bBHXEIvLUonbIKpwdqaxgekJhI6Lw5fBacxO1tkhFRbm+TzF9CJyOvvMKg7yaT+U4yTidkvpPMVVOTCJ03h1kHEnE6IT0dmjeH557zdkM8ywQiw2N+3JVDhStNT3lF47vVBsVF8fex/Zl8Sz+W7szhT3M3momuhu9wBaNZxWM42qU7swquI/SSgbBhAwwcCNddZ43bTkqCOXOsOymXVq3g8cdh7lxYtcqLbfAwM4/I8Jg2kaEAOOrIoNBQSYM7sTf/OP/8YQcOgbiYcIZ3jTltQqxh2E5iIgwdSsQPP0CLFrBtm/Wq9O678OCDpwShSr//Pfz73/D00/D9901X5aZk7ogMjzlWUg7AxMu71ppBoTEeu6oHCT1jmbN6D698a0bSGT4iORnWrydzwgRrRMI770BmpvU1PBxCQuDtt+HDD087tHlz+NOfoHxBMs9HvOyXGRdMIDI8ZlVmHh2jwnjqugvcdtciIgx0nUuBMjOSzrC75OSqbrfM++6zut+SkuDVV62vn3wCU6ZY+06YAJ9/ftop+hy0RtIlFw1B9WTGBX8JRiYQGR6hag1QGNol2u3nHtG9FcEB1jDwAMe5d/kZhsdUC0JV3W6JidYtzhNPWF8TE61nRK+9Bk4n3HILfP31KecY8koSScwhhZNdd/6UccEEIsMjdh0uIreolGEeCESD4qKYfs8QBBhzcXvzjMiwp9qCUGX55MnwyivW12TX0O7f/hYmTbKC0Q03wIIFVee41XlqEKrkLxkXzGAFwyNWZuQBMLSLZ+5WRvaIpW+HSHbnFZ99Z8PwhlWrTg9CNcsvvth6X7nPn/8Mubnwr3/BPffAiRMwZw7p9yZC1umX8JeMC+aOyPCIlRm5xDYPIT6mmceuMaxLDGuzCzhRXuGxaxhGo/3hD7WOgjulPDHRel/d66/DJZfA3r3ws59BYiJ//Ss0q/FfqVkz/8m4YAKR4XaqygrX86GGpvRpiKFdojlR7mTDniMeu4ZhNMbqzLzGJ+pNToYdOyA62hrIsGAB48fD1Kkn74DCwqz3/rI0hG0DkYhEi8hnIlIkIlkiMq6O/SaJSJmIFFZ7da22fYCIpIlIsevrgKZrxflpT/5x9h8pYWi8+58PVVd5/spuQMOwg6U7c7j1zR8bN72g+nOl6dOt/D433QTJyYwfb42We/hha82im2/2XBuamm0DEfAGUAq0AcYDU0Skbx37fqSqEdVe6QAiEgzMAz4AooAZwDxXueEhqzIrnw95NhBFhQfTq01zVphAZNjItMXpKI2YXlBzcMOYMXDZZRAYCLfeWjWoYcwYa8TcggUea0KTs2UgEpFw4BbgOVUtVNWlwHxgQgNPlYA1IOM1VT2hqq8DAlzpzvoap1qZkUdkaCC92jT3+LWGdokmLTOPcpN/zrCBkrIK1u0pwCEQ0JCMIrWNsBOxRtYVFMAvfmFtT04mIQEiI2HePI82pUnZddRcT6BcVbdXK1sPXFHH/qNFJA/YD/xbVV2zw+gLbNBTE5NtcJV/U/0EIjIRmAgQGxtLSkrKOTfCrgoLCz3avpRNxXRt7mDx4kUeu0al5sfLKSqt4L0vk+naIgDwfPu8zZ/b5+ttS84uo6C4jPEXBHOiXOkdHcCxjPWkZFjb62pfp9mzOfb00xSIQI3tfRITifn4Y7b88Y+EzZ7NbhEGDerD3LktGTduOQ5b3k40kKra7gWMBA7UKHsQSKll3z5AeyAAuBQrGN3h2vYcMLvG/jOBSWe6fs+ePdWfJScne+zcB48e17g/fqlvLdrpsWuccr0jp1/Pk+2zA39uny+3ray8Qke+tFDH/HupOp3OWvdpcPteekn1gw9Ug4JU77+/qnjmTFVQXb5cVRcutPazAWC1NuJ3vl1jaSEQWaMsEjhWc0dV3ayq+1S1QlWXA/8Exjb0PIZ7rMqwHsx6av5QTa0jQ+nSKtwMWDC87quN+8nOK+bhhG7uGy06ZAg89pj1YOi//4WNGwGrpy4wEDa+7urSGzLkLCeyN7sGou1AoIj0qFbWH9hUj2MV6zkQrv37yamfin71PI/RCCszcgkLCqBv+5rx33OGdYlmZUYeTqdZGsLwDqdT+U/yLnq2ieCqC9q478SuxfVITrYmDrnmHLVsCY9elMzYj2vJ3OCDbBmIVLUImAs8LyLhIjICGAO8X3NfERkjIlFiGQo8ijVSDiAFqAAeFZEQEXnEVb7Q4404T63IyGNQXBRBAU330RraJZqjJeVsPWBudA3vWLj1ENsOHuOhhG44HG6eO5eYCB9/bH3/zTfWWhDJybywPYlbKuawvYNvByGwaSByeRgIAw4Bs4CHVHWTiIwUkcJq+90O7MTqbnsPeElVZwCoailwI3AXUADcB9zoKjfc7EhxGdsOHvP4sO2aKq+3MsNk4TaanqryRspOOkaFMbpfe89cJDERPv0UHA647z5ISuLI21b+ufnzPXPJpmTbQKSqeap6o6qGq2pnVf3QVb5EVSOq7XeHqsaoNX+ot1pDtKufZ62qDlLVMFUdqKprm7ot54vVWXmoen7+UE0do5rRoWUYKzPNcyKj6aWm57E2u4BfXtGNQE/2BFxzDVx5JezZA7fcQts7Eunf3z+Gcds2EBm+Z2VGHsEBDgZ0atnk1658TqRmCXGjif0nZSetIkK4dVBHz14oORnWrrXuimbMgORkxoyB5cvh8GHPXtrTTCAy3GZFRh79O7UgNCigya89tEs0OYWl7Dpc1OTXNs5fH6/azZIdOVx3YVvPfu4rJ7x+/DHcdhsEBEBSEuPbJ+N0wldfee7STcEEIsMtlu3MYcOeAjpFeS7b9pkMc81eN8O4jaaSlpXPH+duAODj1bs9t2R9zawLEydCURE88AA9nk1idEQyDz+MTy8hbgKRcc7SsvK5951VOBW+3LDPc/8hzyA+phmxzUNYYQYsGE3khy0HqZwxUFbhoSXra0v9c8UV0KMHLFnCDxPnML0wiWHHk316CXETiIxzlpqeS6kr11uFUz3zH/IsRIShXaJZkW6eExlN41hJOQCOhuSUa6jaFtcTsaLNsmW89E5rkpjDEFZVbfbFJcRNIDLO2YWuyauCB/9D1sPwLtEcOFrCnG2lXrkrM84fTqeyZMdhLmjXnMev7sXMB4Z7Zsn6uhbXu/tuCA5m1P63SSGRv3Pq4nq+toS4CUTGOdt+0JrWddcl8Z77D1kPESFBAHyTWd7wdWAMowFS03PJzC3ml5d349eJ3Zv+Mx8bCzffzN2O9wjl+GmbfW0JcROIjHPidCrvp2YxJD6Kv4zp67UgBLC3oBhoxDowhtFAM1dm0yIsiGsvbOudCrz8MgwZQktnPuOCPz1lU9US4snJ1n4+wAQi45ws2n6Y7Lxi7rok3ttV4ZJurQhwpVcJCvBeF6Hh33IKT/DdpgPcMrCjV6YqAFaS08mToX17JneZSrt2VnFMjGsJ8fa+lQzVBCLjnMz4MZPWzUO4pq+X/jKsZlBcFP83qg8AD3uju8Q4L3yatoeyCuWOoZ28V4nKZKhHjtB62xL2fL+FVq2srNxVQciHkqGaQGQ0WmZOESnbDjNuWGeCA+3xUbpzeBwtQ4RN+454uyqGH1JVZq3MZkh8FD2aYAXiM0pMhPfeA8Ax6f/42c+g+Ktk1MeCEJhAZJyDD1KzCHQI44ba58logEMY0jaA5G2HOVpS5u3qGH7mR9cghXHDbPKZv/lmSEiATz/lqWNPMyUviayXfSsIgQlERiMVl5YzZ/Vurr2wLa0jQ71dnVMMaxdIabmT7zcd9HZVDD8za+VuWoQFcd2F7bxdlZOefRZUGfC/yUzhIeYd9a0gBCYQGY00b90+jpaUc/el8d6uymm6tXDQoWUYX2zY5+2qGH4kt/AE3/50gJsHdvDeIIW6OBzQrRuPBEzh4Oxkb9emwUwgMhpMVZmxPJML2kUy2IYDAkSE0f3bs3RHDnlFZukpwz3mrtlLaYWTO2zUFU1yMtx+O9xxB2Rk8MnPpvD71CTKv/etYGQCkdFgM1dks/XAMa7o2YpTV2G3j9H921HuVL756YC3q2L4gbTMPP6TspPebZvT09uDFCpVz0P33HPgdHJJ20xuZQ56a5K13UeYQGQ0SFpWPv837ycA3l2WadvsBX3aRdI1Npwv1pvuOePcpGXlc/vbqeQXl7HrcKE9PvM1k6H26gXDh9Nn1QwWSwIf3DDH2u4jwci2gUhEokXkMxEpEpEsERlXx35PishPInJMRDJE5Mka2zNF5LiIFLpe3zVNC/zTgs1NkHHYDUSE0f3ak5qRy6GjJd6ujuHDZq3MoqzC+tA7vZTU9zS1JUO96y4CtvzEHX3WM22Xa57RqlV1n8NGbBuIgDeAUqANMB6YIiJ9a9lPgLuAKOBa4BERub3GPqNdS4lHqOrVnqy0vzvg+qXu0YzDbjK6fztU4auN+71dFcNHrc3OZ/66/QgQYKfPfG3JUG+7DYKDeSh8BitWwJGBidZ+PsCWgUhEwoFbgOdUtVBVlwLzgQk191XVl1V1jaqWq+o2YB4womlrfH4oLi1n4dZDDOsS7dmMw27SvXVzLmgXabrnjEbZdbiQ+95dRZsWIbx992B+b/fPfHQ0jB7NkB0fIhVlLFrk7QrVn9hx7RYRuRhYpqrNqpU9AVyhqqPPcJwAa4C3VPVNV1kmEIYVdNcCT6rq+lqOnQhMBIiNjR00Z84c9zXIZgoLC4mIiGjwcQuzy3hvcylPDwulZ5TNhq9WU719X6aX8sn2Mv5+eRixzWz5d1eDNfbn5wvs0rb8EicvpJZQ6lSeHRZGm3D3fHY83b6YZcu46NlnuSloHjJqII8+utNj16pNYmJimqoObvCBqmq7FzASOFCj7EEg5SzH/QVYD4RUKxuBFYiaAX8CDgAtz3Senj17qj9LTk5u8DEVFU5NfCVZR72+RJ1Op/sr5UbV25edW6Rxf/xSp6Ts9F6F3KwxPz9f0ZC2fffTAf33wu26OjPPrXVYtO2QDv5/32mvZ/6nG3YXuPXcHv/ZlZaqtmqli9reqr17e/ZStQFWayN+59v1T8RCILJGWSRwrK4DROQRrGdF16vqicpyVV2mqsdVtVhVJwMFWIHOaIBF2w+TfriI+y/rYtsh27XpFN2MHq0j+O/SDHuMdjLcYuriXTz4/mpe+Xa7W9aeUlW2HTjGU59u4O7/ruRwYSlOpWrlYZ8RFATjxjH80HwObM3H4YD4ePsvHR7o7QrUYTsQKCI9VHWHq6w/sKm2nUXkPuAp4HJV3XOWcyvWAAejAaYvzaBNZAi/uMhGqU3qIS0rn4ycIsqdyri3U/nwQRv38Rv1Ulxazr8XWl1OCpS61p5q6M81LTOPuWv3UniinHW7C8jKLT5le4Wzcef1GtcaRV/H3sV1zte5lY95WyeSlWWtLA6uzNyrVtluEIMt74hUtQiYCzwvIuEiMgIYA7xfc18RGQ+8CPxcVdNrbOssIiNEJFhEQl1Du1sByzzfCv+x9cBRlu7M4a5L4m2TZbu+UtNzcbqeg5aaxfL8wmsLdnC0pJyggJN/Tw7vEt2gc6Rl5ZM0NZWZK7KZt24fLcOCeOHGC5l212BCgxz2GiFXX0OGQFIS7/3rCD/Rl7uZUbWpuBi+fNy+axTZ+bfKw1jPdg4Bs4CHVHWTiIwUkcJq+70AxACrqs0VetO1rTkwBcgH9mIN775OVc1vowZ4Z2kmoUEOW2XZrq/hXWOqgqdDxLd+sRin2bzvKNOXZnDH0E7MnngJV/ZujVNhT8Hpy2WfydTFu6hwTYgLELi6b1vuHB7HVX3aMPOB4fYfIVcb1xpFrx+6jWWMYATL6YZ155hAMq8ftO/yEHbtmkNV84AbaylfAkRUe9/lDOfYBPTzSAXPEzmFJ/hs3V7GDupIVHiwt6vTYIPiopj5wHD+NHcDeUWlvvWLxThFhVN5+rONtAwL4o/X9qZls2DevmswN/9nGf/vy81c0TOWls3O/hldviuHBZsP4hCrj77mnc+guCjf/ZwkJvJomzn8++BYKhAm8D4pJDCHJB5tM4dZNgxCYO87IsMGZqZmU1ru5L4RdcZ72xsUF8UdQzuTU1jKnvzisx9g2NKHK7JYt7uA50b1qQo4AQ7hxZsvIr+4jJe+2XrWc2TkFPHQB2voEhvBjPuG+uadz1mM+kcid4Z8QgWB/JbXmEMSd4XMYdQ/7BmEwAQi4wxS03N4a/EuLu7cku6tvT+341wM62L9xbsyI8/LNTEa49DREl7+ZhuXdW/FmAHtT9nWt30L7hsRz6yVu8/48z1SXMb9767CITD97sGM7BHLr/1wSfnx4+HO6Yl8E3YTLTnK8oDLuXN6IuPHe7tmdTOByKhVWlY+E6avpLi0gk17j/r80OdebZsTGRpoApEPSsvKZ8J/V3K8vIIXbryw1ukDv/t5Tzq0DON3c9bx+g/bT/u8rszI5YZ/LyUrr4g37xxEXEx4U1XfK8a3T+aG8IUcDYrm+or5jG+30NtVOiMTiIxaLd+VU5XosXIYqy8LcAhDu0SzwgQin5KWlc8dU39k24FjoJBbx/pSzYIDuefSePbmH+fV73dw+9QfefF/W3h3WQaT5v/E7VNTycorxiFCYICf/9qrlpl71c2TCaQc54032zoTt5//RIzGKndN5POF5Kb1NaxLDBk5RSYbt49QVaYs2kWp6w8i1TNnvq4++bSsQpm6OJ1JX2zm3eVZVRnjbZM921NqLA/R6nd3cZDW5LbubetlIdweiETkdRH5spbySBGZJCIXVCt7TEQ2iogJiDZSXuHki/X7iYtuxu9/3tNvHuYOdc01MXdF9nfoWAn3vbuqanRbfeb1DO8aQ2iQA4dASKCDaXcNJu3Zq3j//qGEBvro3KCGqLlGEXDh4FCmhTxC7K4V1oRXmwYjtw7fFpFuwK+AS2vZPBj4M9ZE1UpvYWVEuBt4x511MRpv/vp9pOcU8eadA7n2Qt/KpHAmfdtHEh4cwIqMXEb3b3/2A4wmlZaVz5e7StnMTqYtyaDoRDl/uaEvF7aPJDUjj+FdY874B1HlUP3U9NxT9h3ZI5aZD55e7ndqWaMoIAA2Xf4QxxdMJmzp0pNrFNlsGLe75xE9BqxX1dW1bLsYOAFsrixQ1eMi8h7wBCYQ2UJ5hZN/LdzJBe0iubpPW29Xx60CAxwMio82AxZsKC0rn/Fvp1JS7oQd2+jaqhlzfjmc7q2tZbkHxdcvc0Jdc4B8em5QfdWRtmfAVa2Y/v19/PqDt5EXXrBdEIJ6ds2JSHcRKROR52uUT3GtjDpYREKAO4EPazl+C/AKEAKUiYiKyKeuzbOBPiJS212UUU1aVj5vJO/06Ai2eev2kZFTxG9/1gOHw/9S8g3rEs32g4Xk1fHQ2/CO1PRcTpRbz3gEuPHijlVByDg3t2W/zI8Mh7Iy+Ne/at8pOdnquvOSegUiVd0JTAMeE5EYABH5P+A+4CbXHdBwoCWwpJZT3AWkA18Al7hev3dtW4eVVfvaxjfD/1X+xfiP77adU7bhyu6P2o637oZ20KddJNf0bXOuVbalYa7nROauyF6GVssVFxLkYET3Vl6sjX9pf8MQXuN37Gp3GUyZAoWFp+6Q7P0cdA0ZJPA8EAA8JSIPYD3vmaCqC1zbh2Mlw91Qy7HrgY7AQlVNdb2yAFTV6do+vJFtOC+kpudQUu7EqXCirHHDqa2hsKl8uqOs1mD2+bp9ZOYW89hVPXxqqYeG6NexJSGBDlZk+PHIKR90pLgMBYa3C/CbwTF2EXR1In+7eA6tD26EggL4739PbqxlgIM31DsQqep+4DXgN8CbwKOqWn0Z0/bAUVWtrc+jLxCMtXpqbQ67jjfqUFp+ciVdBYpOlDX4HN9tOkBphRMFSsqcvP9jZuXigVV3Q33bR/LzPv55NwQQHOhgYOcoc0dkMzN+zKRtZCgPXBRigpAHtLgxkRsr5qKBgfDii1BebpsgBA0fvr0D6znPj6r6Ro1toViDEWozEOv357o6th/HyrRt1CKn8ATv/ZhJzzYR/O6qHgzo1JI3F6Xz1Yb99T6HqrLadQckrtfn6/Zxy5TlrMrM47UfdpCVW8zo/u399m6o0rCu0Wzef5QjxxsezA3323mokCU7crhzeGcC/fC5pIVcx8AAACAASURBVB1cfjkkk8j22/8MBw/CLbfYJghBAwKRiPwMa7j1j8AIEamZ1ToX6xlRbS4Gdqnq0Tq2RwM59a3L+ebP8zdRdKKCN8YN5LdX9WSWa3G3xz5aS/K2Q/U6x5cb9pOWlc89l8ZxS48gPvrlcF665SL2Fhzn1jd/rFpo7LUFp6dH8TdDu0SjCmlZ5q7IDt7/MZPgAAe3++AyI75i2DBr8dZ32j0N7dvD/PlWUjobBCGo/6i5gcBnWAMWEoBsYHKN3bYCwSLSsZZT9KHasO1adAG21acu55tvNx3gqw37+c2V3enRxhpFFBYcwPR7htCzTXMe+iCND37MOuNouiPFZfzli01c1KEFz17fh1HdghnaJYbbhnQm5YlERvY4+WC47DxYPG5g5yiCAxysSDeByNuOlZTxSdoeRvVrR6uIEG9Xx2+FhcHQoXDsy0Vw/LgVlf7zH/jhB29XDahHIBKR7sDXwHfAb1zPgP4C/EJELq+262LX16G1nKYA6C8i14jI8MqRd67ztwR6VjvecDlyvIznPv+JC9pF8quEbqdsiwwN4r37hhITHsyz834642i6F/+3hfziMibffNFpebbCggN47KqevrsqZSOEBgXQv1MLk2HBBuau2UtRaQV3Xxrv7ar4vXvikpm0JYnjH3xqBaGyMhgzxhaZFs4YiESkLVYA2gKMd41wA3gP6w7ob5X7qmomsBIYXcup/g84CHyO1bV3QbVt1wOlWHdcRjUvfrWF3KJS/j62H0G1JGqMiQipyhBQOZpu2c7Dp+zz465cPlq9mwcu68KFHVrUep3KGen+uDZLXYZ2iWbj3iMUnSj3dlXOW06nMuPHTPp3akn/TnX16htukZzMXV8mkcQclockQm4uXHopnDhhPS+qLRg14dyiMwYiVT2gql1VNUFVT1Qrr1DVC1S15iTUKcDNItKsxnl+UtVhqhqmqqKqS6ttvhP4uOby3SISLSKfiUiRiGSJyLja6iiWl0Qk1/V6Sao9bReRASKSJiLFrq8DzvxPYg/vLM3go9W7uaF/+zoDCMDP+7QlxLUUtgIfrd7D2mzrrqikrIJnPttIp+gwHruq5xmvNyguyi/XZqnLsC4xVDiVSV9s8vtnYna1bFcO6YeLuOfSOG9Xxb+5RseVzZzDYkciixdj9dNt2wbNm0OLFqfnoGviuUXuTjb6AbAPeLg+O7uCwpVYXX01vYF1p9QGGA9MEZG+tew3EWtJ8f5Yy4KPBn7pOn8wMM9VryhgBjDPVW5bC7ce5PkvrUdqX2/cf8ZflIPiovjwweE8eU0vnht1AepUbpmynEdnrWHC9BWk5xTx4k0XERYc0FTV9wmVWSM+Wb3nnCYIG403Y3kmrSKC+cVF/pPP0JZcOejCRyUyYABWIEpMhI8/hooKyMyEhARrP/DKsG63BiJVLQfuBeq7HnNb4B5X5oYqIhIO3AI8p6qFrjuo+cCEWs5xN/APVd2jqnuBfwD3uLYlYOXTe01VT6jq61gjl69sUMOa0JHjZTz16UYqZw2VVZx98EDl3cz9l3Xl299dzlUXtGH++v2syswnQIRmwe5OKej71u8uAKy7yBNlTr7ddMC7FTrPZOcW88PWQ9wxtDMhgeaPJI/6wx+qAkrr1pCSAg4HxN+byIJHPofQUPjkExg40Gtzi6RyQqOdiMjFwDJVbVat7AngClUdXWPfI8DVqrrC9X4wkKyqzUXkd65t11Xb/0vX9n/UOM9ErLsrmjVrNqi4uL6x1H0kKITWSf+PkHY9QZ0gDtRZwcHZz1C6b2u9zxM5/FZajrwTcQSgzgoKlnzA0dSPPVhz3xPcvjdtbv8rEhgECDgrKNz4PUeWzyGgeQyhnS+iJHtjg/7djfqLGfU44RdcweG5L3B810pvV+c8cQfwX6wpn5WKuIbr+ZJFHMfqghoLpDT+ImmqOrihB9n1T+UIoOacoyNAbVkQI1zbqu8X4XpOVHNbnedR1anAVIBevXrptm1NO5r8RHkFD8xYzbKdObwxbiCtI0NPpq3/x5YGnSstK5/x01IpK3cSFBLE3NlvMSjuZBKMlJQUEhIS3NwC+6hv+9Ky8klNz6VH6wiW7Mjho+BgWg68DhBUleBAhy0Hb/j6z29Fei63TU0FoPMdk075N/b1tp2NN9sXHw9ZWTVLw9kal0Lgtb+i+VtvQUwMyVu2QGxso67R2Mnwdg1EhUBkjbJIrOSoZ9s3EihUVRWRhpzHa8ornPzmw7Us2ZHD38f24zpXn3ljfwHWtS6LcarqSwNc3bctDyV04+GZaazbbf3tUjmnyvz7uddHq3dXfW/+jZtOdnbt5V2zkuHTT+Huu2HGDKuLbtMmiKz5q9MlOdl6nlTHshONYdeVUbcDgSLSo1pZf2BTLftucm2rbb9NQD85NUz3q+M8XrE6M48x/17Gd5sPMml0H24d3Mkt5z3fRsG5Q/uWYTw3qi+VWWbOhzlV3nDoqDUA93yZt2YXnWtJXJFAMh87XM+E3n0X7r8f9uyxglFJyekHeGg0nS0DkaoWYa3k+ryIhIvICGAM8H4tu78H/F5EOohIe+Bx4F3XthSgAnhUREJE5BFX+UJP1r++0rLyuX1qKpv2HyXQIVzU0cyl8LZBcVE8d30fAO6/rIsJ5G5WXFpOWlY+P+/T5ryat2YHf/0rNKs2sSaBZOaQxNqnqg1MmDYNxo2DXbtg8GArOWolDw5ksGUgcnkYKxHqIWAW8JCqbhKRka4ut0pvYa1ztBH4CfjKVYYrC8SNWOshFWCtn3RjHRnCm1xqei7lTmuwiKr6fWodX3HPiHh6t23Ogs2HsONgHl/2/eaDHC+r4IHLupg79iY2fjxMnQpxcSeD0Jqn5nDVX2sElZkz4aabrO65Sy4Bp9Pjo+lsG4hUNU9Vb1TVcFXtrKofusqXqGpEtf1UVf+gqtGu1x+02m8PVV2rqoNck2kHqupab7SnNn3bW32wgumisBMR4cGRXdl28BiLd5hcvO70xfp9tGsRypB6Lv1tuNf48da0oTfuXkUScyi/rI6gMncuXHMNrF5tZWDw8JBu2wai80FJmZUx6Y6hnU0Xhc2M7t+eNpEhTFuS7u2q+I2C4lIWbT/M6P7t/XIZel/Sfeof2BiTyAcfnGGnr7+2ngWtWAEDBnh0XpEJRF6Ump5LaJCDSTf0NUHIZoIDHdx9aTxLduSwZX9dq5c0TFpW/hmzpPu7r386QFmFckN/swamtwUHw223weefw9G6Pt4pKZCRYc2CXbAAnnmm7hOeY146E4i8KDU9l8Fx0QQHmh+DHY0fGkdYUADTlmSc87nSsvK54+3UM2ZJ93fz1u2la2x4VZe04V133mkNjPustnTT1Z8JvfceBARYK7vWFmzcMJLO/Ab0kryiUrYeOMbwrqav3K5aNAsiaXBH5q/fy8GjtQxlbYDP1+6ltNyJU60u2a821n91XX9w4EgJKzLyuOE8WAHYVwwfDl27cnr3XM2BCddcY0UrhwP++Ed46626920kE4i8ZGWGNULODFCwt/su60K5U5mxPLPR5zh87ARfbdhftUQ7WNnVH5u9lvnr954X3XVfbtiHKqZbzkZErLuiH36AfftchXUFltGjYfZs66Bf/cqKXm4cSWfXzAp+LzU9j7CgAPqZuUO2FhcTzjV92jJzRTaPXNm9wQlkS8ud/HrmGorLyvn7rf04ePQEfdo1Z0VGPu8sy+DzddZvgJBABx8+6L8DVuat28dFHVrQNTbi7DsbTWb8eHj+eZg1Cx5/nKpM3bUGlltvheJiuPdemDjRmpT08cduGcRg7oi8JDU9l8HxUeb5kA948PIuHDlexm9mrW3wncsLX21mZWYeL93Sj7GDOvHrxO4k9m7DU9f15oGRXarukE6UO1mw5aD7K28D6YcL2bj3CGMGmLshu+nZ01qaqKp7rlqm7lrdfTfcc4+13PjYsW4bSWd+C3rByedDplvONwgi8MOWQ9z65nKe+Wwja7PzKa9wnnEk3JzVu3nvxyweuKwLYwZ0OG37lb3bEBLkqEop9EnabjJyijzdmCY3f/0+RGBUPxOI7KhXL1i3zrU0RLw1n7VOycnwxRfw9NNWfjo3LTNuuua84OTzITNQwRekpuciWGsXORVmrshm5opswoICOFFegSoEBTh4594hjOjeirSsfD5fu5dZq7K5tFsMT13Xu9bzVk9OGxsRzN++2cbYKcuZcd/QM67K60vSMvOY8WMmfdo1p22L0LPubzStmTOtpYgAVK3s3BMnWu/Hj6+xc81nQlddZZ4R+bLK50MXdTDPh3zB8K4xBAc6rGU1Ah38Z/xAjpc6eXvJrqpM3aUVTu6ctoLOMc3Yk3+cCqciwIOXdyUwoO6Oh+oZwAfHRzNh+kpun5rKE1f3pKi0wqezp6dl5XPHtBWUljspLLFyzPlqW/zVM89YvWzVFRdb5acEotoGJiQmWu/NqDnfZJ4P+ZbKO5fKJJ1X9m7D9f3a8dyovoQGOQgQCA4Qxg7uiAAVrvyBIrB5X/0nw3aNjeDThy4lqlkQk77YzCvf+vaco9T0XMrKrewhTqfJpWhHdS0NkZ1t3S3Fx8OVkkzuVUksmFhLsKkejM6hm87cETWxyudDo80wVp9S/c6lelnNdZ9OWZSwEfkD27YI5caLO/CvhTtRrFF3vrpez/CuMYhYXT4ml6I9de5c22J51s/s3nuhrAySWMVY5xxWvpbI1D61dNlVBqNVqxpdD/MneRNbkW6eD/mTmus+1bx7akwASejVmhDX3bIqDPbBIATQr2MLAh0OBnZuaXIp2lTNpSEAQkOtFEBlZdb7v/MHUkis6rKrVWLiOS2UZwJRE0tNzzXzh/zcuS5KOCguig8fHM5NA9qjwMKth9xbwSaydf8xSiuc3DvCrOtkV9WXhhCxvk6bdjII1VRXV965Ml1zTSw1PY/B8VEEneEBtmFUdgWGhQTy1uJ0Lu8Zy4jurbxdrQZJy8oDGr/kvdE0xo8/vbvtmWdq77KrbZVXdzC/DZtQbuEJth0084eM+nv2+gvoGhvO43PWk19ki/Uc6211Vj7tWoTSvmWYt6tiNFBtXXYhIVa5J5hA1IRWZlh/IZpAZNRXs+BAXr/9YnKLTvD0Zxt9asXYNWa4ts+q2WUXEACtWllZfjzBdoFIRKJF5DMRKRKRLBEZd4Z9nxSRn0TkmIhkiMiTNbZnishxESl0vb7zfAvqdvL5kH9MVjSaxoUdWvDE1b34+qcD/P3brXy5q9T2Q7r3FRxn35ESE4h8WOVqrk6ntW7R3r3w6queuZbtAhHwBlAKtAHGA1NEpG8d+wpwFxAFXAs8IiK319hntKpGuF5Xe6rS9ZG89TBtIkPYsOeIN6th+KAHR3blog6R/CclnU93lNl+flFl3QbHmdGh/mDUKLj5ZitBasa5L891GlsFIhEJB24BnlPVQlVdCswHJtS2v6q+rKprVLVcVbcB84ARTVfj+kveepDs/GKycott/0vEsB+HQ6oGKyhQ5ppfZFdpWfmEBQXQu11zb1fFcJN//tO6O+rTp5556RrAbqPmegLlqrq9Wtl64IqzHSjWalsjgbdqbJopIg5gLfCkqq6v4/iJwESA2NhYUlJSGl77M3h1tZVHQ4HSMiezFqziWLdgt16jvgoLC93ePjvx1/a1Kqlw5bxTHCKEFGSRkrLH29Wq1aKfjhPXHJYtWdyg4/z1Z1fJl9u3YEFrKip6U15u3b9kZcH991ewZcs2rrrq3KYY2C0QRQA1c6IcAerzZ9UkrDu8d6qVjQfWYHXh/Rb4VkR6q2pBzYNVdSowFaBXr16akJDQ0LrXqaSsgt2Lf8AhTgRrlvkdVw3xWv95SkoK7myf3fhr+xKA8uhd/O3rrTx5bW8euLybt6tUq+LScrK/+46HruhGQkKvBh3rrz+7Sr7cvnvugfLyU8tOnAjggw/68MILfc7p3E3aNSciKSKidbyWAoVAzQXtI4FjZznvI1jPiq5X1ROV5aq6TFWPq2qxqk4GCrDumprUZ2v3UlBcxv+N6nNOM+4N474RXWgeBOtt/Jxx/e4jVDjVfMb9zJny0p2rJr0jUtWEM213PSMKFJEeqrrDVdwf2HSGY+4DngIuV9Wz9VMoJ1drbhIVTuWtRbvo17EFd18aj9WDaBiNExzoYFi7QL7ffJAjx8toERbk7SqdZk229fzz4s4me4g/qSsvnTsmudpqsIKqFgFzgedFJFxERgBjgPdr219ExgMvAj9X1fQa2zqLyAgRCRaRUNfQ7lbAMs+24lTf/HSAzNxifnVFNxOEDLcY0SGQ0nInX23Y7+2q1Gp1Zh49WkfQspl3noEanlHbJNdmzdwzydVWgcjlYSAMOATMAh5S1U0AIjJSRAqr7fsCEAOsqjZX6E3XtubAFCAf2Is1vPs6VW2yoUaqypuLdtGlVTjX9G3bVJc1/Fx8pIPurSOYu8Z+AxWcTmVNdoHplvNDteWlmzq1lmzcjWC3wQqoah5wYx3blmANaKh83+UM59kE9HN7BRtg2c5cNu49wt9uvogAh7kbMtxDRLh5YAde/mYbWblFxMWEe7tKVdJzCjlyvMwEIj9VW146d7DjHZHfmLJoJ62bh3DTwA7erorhZ24c0AERmLtmr7ercorVmdbzIROIjIYwgchDNuwpYNnOXO6/rAshgQHero7hZ9q3DOPSbjHMXbvHVvnn0rLyiQ4Ppksr+9ylGfZnApGHvLloF81DAxk3zEN5043z3s0Xd2R33nFW2yhLR1pWPgM7R5mBOUaDmEDkAV+s38v/Nh7g6j5taB5qv+G1hn+49sK2NAsOsM2ghbyiUtJziky3nNFgJhC5WVpWPo99ZGUR+nLDfpNTzvCY8JBArr2wLV9u2E9JWYW3q8OaLPN8yGgcE4jcbOmOw1Q4rT778gp7J6Y0fN8tAztyrKScJz9Z7/U/elZn5RMUIGaZE6PBTCByN1ffuEOsnHJmETzDk4IDrf/CX6zf7/Ws7ou2HaJVRAib9tVMF2kYZ2YCkZtt2F1ATHgwv/95T5NTzvC4lRl5VTmrvLk0xI+7cthy4BgHjpR4PSAavscEIjfKKypl0fbDjB3UkUeu7GGCkOFxw7vGVN0ViYjX7sCnLbVWS/OFtZIM+zGByI2+2rifcqcyZoCZwGo0jUFxUXz44HA6RYURHR7ExZ2aPtFoabmTddkFiECA6ZI2GsF2KX582by1e+nZJoILzKqURhMaFBfFE9f04rez17F8Vy6X9WjVpNefu2YPuUWlPDfqAkrKnAzvGmN6A4wGMYHITXbnFbM6K58nr+llJvMZTe6avm1pERbE7FXZTRqIyiuc/CfFWubkvhFdzGffaBTTNecm89fvA2DMgPZerolxPgoNCuCmizvw3aaD5BWVNtl1v9iwj+y8Yh5J7G6CkNFoJhC5gary+dq9DImPomNUs7MfYBgecPvQTpRWOJss04LTqfx74U56t23OVRe0aZJrGv7JBCI32Lz/KDsOFXKDGaRgeFHvtpEM6NSS2at2N0ki1K9/OsCuw0U8cmV3HGaZE+McmEDkBvPW7SPQIVx/UTtvV8U4z90xtBM7DxV6fB6PqvKvhTvoGhvOdReaz71xbkwgOkcVTmX+un1c0TOW6HCzNLLhXaP6tSc8OIDZq3Z79Do/bDnE1gPH+HVCd7Poo3HObBeIRCRaRD4TkSIRyRKRcWfYd5KIlFVbJrxQRLpW2z5ARNJEpNj1dYC767siI5cDR0sYc7HpljO8LzwkkBsGtOfLDfs4WlLmkWukZebx7LyfaN08mBvM4BzDDWwXiIA3gFKgDTAemCIifc+w/0eqGlHtlQ4gIsHAPOADIAqYAcxzlbvN/HX7CA8O4OfmYa1hE7cN6UxJmZP56/a5/dxpWfnc8fYKDhwpIb+4jA17jrj9Gsb5x1aBSETCgVuA51S1UFWXAvOBCY04XQLWPKnXVPWEqr4OCHClu+qbmp7DZ2v3Mjg+irBgswqrYQ/9O7agd9vmzF6V7fZzp6bnUlrhBKxRcyaVj+EOdpvQ2hMoV9Xt1crWA1ec4ZjRIpIH7Af+rapTXOV9gQ166vChDa7yb2qeREQmAhMBYmNjSUlJOWNFd+ZX8LeVJZQrLNuRw7TPfqB7lG8Eo8LCwrO2z5eZ9sGgqDJmbinl/v98yyXtAt322QzML6/6PkAgpCCLlBT3DRc3P7vzk90CUQRQM4f8EaCunDlzgKnAQWAY8KmIFKjqLNe5avYb1HkuVZ3qOhe9evXShISEM1b0p4U7KHfFSwVOtIwjIaH7GY+xi5SUFM7WPl9m2geObYeZuWUlP2SXs2y/022Z4MPSc2FlKjcOaM+ES+LdnsrH/OzOT00aiEQkhbrvbpYBvwEia5RHAsdqO0BVN1d7u1xE/gmMBWYBhQ05V0O51r4z6w4ZtrRx38m/wSqzYbsjaCzecZgAh/D8jRcSGRp0TudyOp3k5ORQUFBARYW1wmyLFi3YsmXLOdfTrny5fQEBAbRs2ZJWrVrhcLj3qU6TBiJVTTjTdtczokAR6aGqO1zF/YFN9b0EVC3Psgl4XESkWvdcP6zBEOekwqnMX7+PjlFh3D6kE5d0a2WSPBq2MrxrDMEBDkornDgc7lseYsmOHC7u1PKcgxDAnj17EBHi4+MJCgpCRDh27BjNm/tv0mBfbZ+qUlZWxsGDB9mzZw+dO3d26/ltNVhBVYuAucDzIhIuIiOAMcD7te0vImNEJEosQ4FHsUbKAaQAFcCjIhIiIo+4yheeaz2/3LCPnYcK+dN1F5h1hwxbGhQXxcwHh9E8JJA+7SLd8hnNKypl494jXN4z1g01hKKiIjp06EBwcLDJU2dzIkJwcDAdOnSgqKjI7ee3VSByeRgIAw5hdbE9pKqbAERkpIgUVtv3dmAnVnfbe8BLqjoDQFVLgRuBu4AC4D7gRld5o5VXOHltwQ56t23OdRe2PZdTGYZHDYmP5t4R8WzYe4R9BcfP+XxLd+agCiPdmN3b3V08hmd56udlt8EKqGoeVgCpbdsSrEEIle/vOMu51gKD3Fm/z9ftIyOniLcmDDL5tQzbGzuoE68v3MncNXt45Moe53SuxdsP0yIsiH4dm37xPcO/mT9HGqCswsnrP+ygb/tIru5jJrAa9tc5phnDukTzSdqec0qEqqos2XGYy7q3Mil9DLczgagB5q7ZQ3ZeMb//eU/Tp234jFsHdyIzt5hVmY1PhLr9YCEHj57g8p5Nu/qr0TCTJk3iwgsv9HY1GswEonoqLXfy+g876d+pJVf2bu3t6hhGvf3ioraEBwfw8erGJ0JdvP0wACN7uGeggj9Ys2YNAQEBjBgxokHHJSQk8Mgjj5x9x/OICUT19Mp329hbcJwb+rczd0OGT2kWHMiofu35auN+ik6Un/2AWizecZjurSNo3zLMzbU7dzNnQnw8OBzW15kzm+a606ZN4+GHH+ann37y2blBdmECUT2sSM9l6uJ0AP7+7TaPr/ViGO526+COFJdW8NXG/Q0+tqSsgpUZeVxuw7uhmTNh4kTIygJV6+vEiZ4PRsePH+fDDz9k4sSJjB07lunTp5+yPTU1lSuvvJLw8HBatGjBlVdeyb59+/jVr37FokWLeOONNxARRITMzExSUlIQEXJycqrOkZmZiYiwevVqACoqKrj//vvp0qULYWFh9OjRg5dffhmn0+nZxjYB242as6N3l2dWfe/OWeqG0VQGxUXRtVU4n6zeQ9LgTg06dmVGHifKnYxsgudDf/xjCJs3n32/SqmpcOLEqWXFxXD//fD22/U7x4AB8Npr9b8mwCeffEJcXBwXXXQREyZMICkpicmTJxMUFMT69etJTExkwoQJvPrqq4SEhLB48WLKy8t56aWXyMjIoHfv3rz44ouAldsyMzPzrNd0Op106NCBOXPmEBsby8qVK5k4cSIxMTHcf//9DWuAzZhAdBZOp7JhTwGCSedj+C4RYezgjrz8zTYyc4qIbxVe72MXbz9McICD4V3s97mvGYTOVu4u06dPZ8IEa1GAK664gmbNmjFv3jzGjh3Lyy+/zIABA5g6dWrV/hdccAFgZVYIDg6mWbNmtG3bsHmIQUFBPP/881Xv4+PjWbNmDbNmzTKByN8t2HKQvQUlPHZVD4ICrCBk7oYMX3TLwI78/ZttPDV3A09e07ven+MlO3IY0qVpljp56aUTNG9e/yXD4uOt7ria4uLAU0mud+7cydKlS/nwww8BK8iPHz+e6dOnM3bsWNauXctNN93kkWu/+eabTJs2jaysLI4fP05ZWRlxcXEeuVZTMoHoDFSVNxftolN0GI8kdicwwDxSM3zXnvzjiEBqeh7jp6XWKyP3gSMlbDt4jJsH9m6iWjbMX/9qPRMqLj5Z1qyZVe4p06ZNo6Ki4pR8a5VztHbvbtzIxMqMBdXnepWVnbrC7kcffcRjjz3GK6+8wqWXXkpkZCRvvPEGn332WaOuaScmEJ3Bqsx81mQX8PyYviYIGT4vNT2Xyt9zJ8rq96xzyQ57D9seP976+swzkJ0NnTtbQaiy3N3Ky8uZMWMGkydPZtSoUadsmzBhAu+88w4XX3wxCxfWndIyODi4Ktt4pdhY6993//79Vd+vW7fulH2WLl3KsGHDThn6vWvXrnNqj12YQHQGby7aRXR4MLcOatjDXcOwo+FdYwgJclBS5kSBuJhmZz1m8Y4cWkWEcEE7+2aMHj/ec4Gnpq+++oqcnBwefPBBYmJOfWZ2++238+abbzJ37lwuueQSJk6cyK9//WtCQ0NZsmQJV199NVFRUcTHx7Ny5UoyMzOJiIggOjqa7t2706lTJyZNmsTf/vY3MjMzeeGFF045f8+ePXn33Xf5+uuv6d69O7Nnz2bRokVERfn+owLzZ34dth04xsKth7jn0nizDLjhFwbFRTHzgeE8lNCNZsEBvLc8C6ez7rQ/TqeSkDIX/gAAEatJREFUsvUgbZqHsCa7oAlral/Tp08nMTHxtCAEcOutt5KZmcnhw4dZsGABW7duZfjw4QwbNozZs2cTFGQtnfHEE08QHBxMnz59iI2NJTs7m6CgIGbPnk16ejr9+/fnz3/+c9Woukq//OUvSUpKYty4cQwZMoTMzEwef/zxJmm3p8m55J/yV7169dLrn/+IbzYdYPlTV9KyWf0fnvoCf18l0rTv7D5evZsnP9nA82P6ctcl8bXu89evNvP2kgwECAlyuG2V10pbtmypGk1WyVfX66kvf2hfbT+3SiKSpqqDG3pOc0dUi3InzFu/jzuGdva7IGQYAGMHdeTynrH87eut7M4rPmWbqvLagu28vSTDes/J+XOG4QkmENXiaKkiwP2XdfF2VQzDI0SEyTdfhEOEp+ZuqBqtVV7h5OnPNvLagh0k9IwlNMhBgJk/Z3iYGaxQi6OlyvXdW9kyr5ZhuEuHlmH86Re9eeazn/ho1W7GDOjAb2atZcGWg/w6sRtPXN2LNdkFpKbnmvlzhkeZQFSHH9NzScvKN//5DL92x5DOfLl+P89/sZl//rCD/UdKTnluNCguyvwfMDzOdl1zIhItIp+JSJGIZInIuDPs+7WIFFZ7lYrIxmrbM0XkeLXt39W3HuUVpk/c8H8Oh3Dn8M4Ul1Ww/0gJQQFC3/YtvF0t4zxjxzuiN4BSoA0wAPhKRNar6qaaO6rqddXfi0gKUHMm2WhVXdDQSpg+ceN8kZlbjEPAqdaQbZPU12hqtrojEpFw4BbgOVUtVNWlwHxgQj2OjQdGAu+daz2iQsTtQ1UNw66Gd40hONAMSjC8x253RD2BclXdXq1sPXBFPY69C1iiqpk1ymeKiANYCzypquvPdqIWIWKCkHHeqJzoagYlGN5it0AUARytUXYEqM8MsLuAF2qUjQfWAAL8FvhWRHqr6mnTxEVkIjARrLxPKZ5K3WsDhYWFpn0+zFPt6ytwLGMPKRluP3WtWrRowbFjx04pq6ioOK3Mn/hD+0pKStz/+VPVJnv9//buPbiqejvg+HcdEhKeF3kk3OR6CeHVAPKIgloexgEGaPTSFl9gGagiNjxGpHZkJgoIDojDI/ZWrCBXItzhluFSOmjxDyGUR8VJQG5piKLAxJpHe7UQAgFCYfWPvXNIDgGSQ5J99nF9ZvbknN9v7521cs7JOmfv3/ltYB/O9+PqWw4CQ4GqkG3+Fth1h/2OBC4A7e+w3lc454xuG2ffvn01muXl5XkdQrOy/PzhxIkTN7WdP3++YRuvXKm6d+/t19m711kvgjQ4vwhW3+NWAyjQMGpDi54jUtUMVZVbLCOBk0CMiPSptdlg4KaBCiGmAztU9cKdQsD5dGSM8bNhw+CppyAvr/7+vDynf9iwZvn1M2bMCF7qu/YSOmO2n4gI27dv9+R3R9RgBVW9COwAlopIOxEZAUwCNt9qGxFpAzwFbApp/6WIjBCR1iISLyJ/B3QFDjVbAsaYlvHoo7BtW/3FqKYIbdvmrNdMxo4dS1lZWZ1l4MCBjd5PdXV1M0TnLxFViFyzgTbA/wBbgSx1h26LyCgRCf3U8+fAOSD0rVEH4D3gLFACTAAmqqp9OciYaFBfMWqhIgQQFxdH9+7d6ywxMTHs37+fBx98kPj4eBITE3n55ZfrFJuMjAyysrJ45ZVX6NatGyNGjADgxIkTZGZm0qFDBxISEpgyZQrl5eV1fmdubi733XcfcXFxJCYmMn369GDfmjVrGDRoEO3atSM5OZmZM2dy7tyN0+EVFRVMmzaNhIQE4uPjSU1NJScnB3AuOw7ODOIiErzfUiJtsAKq+r84xaW+vgM4Axpqt23FKVih6xYCg5ojRmNM84h79VU4caJxGyUlwfjx8POfQ1kZpKXBG284S0MMGQLuP+S7VVJSwsSJE5k2bRqbNm3i1KlTzJw5k0AgwOrVq4PrbdmyhVmzZnHgwAFUlbKyMkaPHs3zzz/PqlWruHr1KtnZ2UyaNInPP/+cQCDA+++/z0svvcTy5cvJzMzkwoULdS7AFwgEyMnJITU1leLiYubNm8e8efPYvNk5oPTaa69x/PhxPv74YxITEzlz5gx//KNz4cP8/HwSEhLYsGEDjz32GK1ateylbyKuEBljTKPcc49ThGou0dpCF4r79NNPad/+xvviUaNGkZ6eTlJSEuvWrSMQCJCWlsZbb73Fiy++yLJly4Lr9uzZs05hWrRoEYMHD2blypXBto8++ojOnTtTUFDA8OHDWbZsGfPnz2fBggXBde6///7g7fnz5wdvp6Sk8PbbbzNp0iRyc3MJBAIUFxeTnp7O8OHDAejRo0dw/Zqrwnbq1Inu3bs3xZ+nUawQGWMixpWVK2nd2Ov11ByOe/11eO89WLy42Q/LAYwePZr169cH77dp04Z58+bx0EMPEQjcOOsxcuRIqqur+fbbb+nZ05nRv3YBAThy5Aj79++vU9hqnDp1ipSUFEpKShgzZswt49m7dy8rVqygqKiIiooKrl27RnV1NeXl5SQlJZGVlcUTTzzBkSNHGDduHI8//jiPPNKQr2g2PytExhj/Cj0n9OijLXaOqG3btvTu3bvB64vcGLDbrl27On3Xr18nMzOTVatW3bRdYmIily5duu2+i4uLyczM5IUXXmDp0qV06dKFo0ePMmXKlOD5qYkTJ1JcXMzu3bvZs2cPmZmZPPnkk3z44YcNzqG5ROJgBWOMubP6BibcbjRdC0hLS+Pw4cNcv3492Hbw4EFat25Nr169brldeno6hYWF9OjRg969e9dZagYvJCcns2fPnnq3LygooLq6mrVr1/Lwww/Tt29fSktLb1qva9euwfNXGzduJDc3lytXrgAQGxvLtWvX7vIvEB4rRMYY/7nd6DgPi9Hs2bMpLS1l9uzZFBUV8cknn7Bw4ULmzp1L27Ztb7ndnDlzqKio4Omnn+aLL77g9OnTfPbZZ8yaNSs4E0N2djY5OTmsXbuWkydPcuzYseB5pj59+nD9+nVycnI4c+YMW7duDY6Iq7Fo0SJ27tzJN998Q1FRETt27CA1NZW4uDjAOa+0Z88eysvLOXv2bDP9hepnhcgY4z/5+bc//FZTjPLzWzSs5ORkdu/ezZdffsmQIUN47rnnmDJlCsuXL7/tdklJSRw6dIhAIMCECRMYMGAAc+bMIS4uLlgosrKyePfdd9mwYQMDBw5kwoQJFBY63/UfNGgQ77zzDmvWrKF///588MEHNx3mi4uLIzs7m8GDBzNixAgqKyvZtWtXsH/16tXk5eVx7733MnTo0Cb+y9yeqHuJYHNDv3799Ouvv/Y6jGazb98+MjIyvA6j2Vh+/lBUVERaWlqdtsrKSjo0drCCj0RDfvU9bjVE5IiqPtDYfdonImOMMZ6yQmSMMcZTVoiMMcZ4ygqRMcYYT1khMsZ4xgZL+UtzPV5WiIwxnoiNjb3jjAEmsly6dInY2Ngm368VImOMJxISEigpKaGqqso+GUU4VaWqqoqSkhISEhKafP8215wxxhMdO3YEoLS0lKtXrwJw+fJl4uPjvQyrWfk5v9jYWBITE4OPW1OyQmSM8UzHjh3r/GPbt29fi3+rvyVFe37hskNzxhhjPGWFyBhjjKcirhCJyFwRKRCRKyKyqQHrvywi5SJyXkR+IyJxtfpSRCRPRKpE5CsRGduswRtjjGm0iCtEQCnwJvCbO60oIuOBhcAYoAeQCtS+UP1W4EugC5ANbBeRbk0dsDHGmPBFXCFS1R2quhP4sQGrTwc2qmqhqp4FlgEzAESkL5AOLFbVS6r6e+A4MLl5IjfGGBMOv4+aGwD8S637fwASRaSL23daVStD+gfUtyMRmQXMcu9eEZH/bIZ4I0VX4Aevg2hGlp9/RXNuEP359QtnI78XovZARa37Nbc71NNX059c345UdT2wHkBECsK5poZfWH7+Fs35RXNu8NPIL5ztWvTQnIjsExG9xXIwjF1eAGp/u6rmdmU9fTX9lRhjjIkYLVqIVDVDVeUWy8gwdlkIDK51fzDw36r6o9uXKiIdQvoLw8/AGGNMU4u4wQoiEiMi8UAroJWIxIvIrQ4hfgQ8LyL9RaQT8BqwCUBVTwLHgMXuPv4CGAT8vgFhrL/bPCKc5edv0ZxfNOcGll+9JNImGxSRJcDikOY3VHWJiPwSOAH0V9Xv3PUXAK8CbXCKzN+o6hW3LwWnMD0IfAfMUdXPmj8LY4wxDRVxhcgYY8xPS8QdmjPGGPPTYoXIGGOMp6wQ1SIinUXkn0XkoogUi8hUr2O6G7ebt09Exrjz71W58/H18CjMsIhInIhsdB+nShE5JiITa/X7Oj8AEdkiImXuPIonRWRmrT7f5wcgIn1E5LKIbKnVNtV9XC+KyE4R6exljOFyv65yWUQuuMvXtfp8n6OIPCMiRW4Op0RklNve6OemFaK63gWqgUTgWeA9Eal3JgafqHfePhHpCuwAXgc6AwXAP7V4dHcnBvgv4BHgZzgjJre5E91GQ34AK4AUVe0I/Ap4U0Tuj6L8wHnN5dfccV9v7wPTcF6HVcA6b0JrEnNVtb279IPoyFFExgErgb/GmUBgNHA63OemDVZwiUg74Cww0B36jYhsBkpUdaGnwd0lEXkT+IWqznDvzwJmqOqfuvfb4Uw7MlRVv/Is0LskIv+BM+ltF6IsPxHpB+wDXgI6EQX5icgzwF/ijITtrap/JSLLcYrvVHedXkAR0CVkuq6IJyL7gC2q+kFIu+9zFJF/x5nnc2NIe1j/W+wT0Q19gf+rKUKuW85N53MDcHIDQFUvAqfwca4ikojzGBYSRfmJyDoRqQK+AsqAfyUK8hORjsBSYEFIV2hup3COUvRtueia1AoR+UFEDolIhtvm6xxFpBXwANBNRL4Vke9F5B9EpA1hPjetEN3QHjgf0laB87Ez2txqHj5f5ioiscBvgVz3XVfU5Keqs3HiHoVzyOMK0ZHfMpx31N+HtEdDbjVexbk0TTLOFz13uZ9+/J5jIhALPIHzvBwCDMU5PB5WblaIbvgpzU0XNbmKSADYjPOOcq7bHDX5AajqNVU9CPwCyMLn+YnIEGAssLaebl/nVpuqfqGqlap6RVVzgUPAn+H/HC+5P3+tqmWq+gOwhrvIzQrRDSeBGBHpU6stWuemqzNHn3sctxc+y1VEBNiI8w5tsqpedbuiIr96xHAjDz/nlwGkAN+JSDnwCjBZRI5yc26pQBzO69PvFBB8nqN77bfvcfIJNrs/w3tuqqot7gL8Dueqru2AETgfKQd4Hddd5BMDxOOMvtrs3o4Burm5TXbbVgKHvY43jPz+ETgMtA9p931+QALwDM6hjlbAeOAizug5X+cHtAW611pWAdvdvAbgHCIf5b4OtwC/8zrmMHLs5D5mNa+5Z93Hr2805Ihzfi/ffZ7eAxzAOdwa1nPT84QiacEZbrjTfcJ8B0z1Oqa7zGcJzjuV2ssSt28szgnwSzijsVK8jreRufVw87mMczigZnk2SvLrBvwbcM79p3UceKFWv6/zC8l1Cc7ospr7U93X30WcC1929jrGMB+/fJxDUudw3jCNi5Yccc4RrXNzKwf+Hoh3+xr93LTh28YYYzxl54iMMcZ4ygqRMcYYT1khMsYY4ykrRMYYYzxlhcgYY4ynrBAZY4zxlBUiY4wxnrJCZIxPiEhHEVkiImlex2JMU7JCZIx/PAAsxvlWuzFRwwqRMf4xFOcyECe8DsSYpmRT/BjjAyJSBPxJSPMOVZ3sRTzGNCUrRMb4gIgMw5kdvhBY7jaXqWqxd1EZ0zRivA7AGNMgf8C5MN6vVfWw18EY05TsHJEx/jAAaA0c9ToQY5qaFSJj/CEd5/pLx7wOxJimZoXIGH8YCpxS1fNeB2JMU7NCZIw/9MeGbZsoZYMVjPGHc0C6iIwHKoBvVPVHj2MypknY8G1jfEBEBgIbgUFAPDBKVQ96G5UxTcMKkTHGGE/ZOSJjjDGeskJkjDHGU1aIjDHGeMoKkTHGGE9ZITLGGOMpK0TGGGM8ZYXIGGOMp6wQGWOM8dT/A6cXj8rX+La7AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n", "plt.show()" ] }, { "cell_type": "markdown", "source": [ "This model works nicely: the MSE for the next 10 time steps is about 0.008. That’s much better than predicted one by one. But we can still do better: indeed, instead of training the model to forecast the next 10 values only at the\n", "very last time step, **we can train it to forecast the next 10 values at each and every time step.** In other words, we can turn this sequence-to-vector RNN into a sequence-to-sequence RNN. The advantage of this technique is that the loss will contain a term for the output of the RNN at each and every time step, not just the output at the last time step. This means there will be many more error gradients flowing through the model, and they won’t have to flow only through time; they will also flow from the output of each time step. This will both stabilize and speed up training.\n", "\n", "Now let's create an RNN that predicts the next 10 steps at each time step. That is, instead of just forecasting time steps 50 to 59 based on time steps 0 to 49, it will forecast time steps 1 to 10 at time step 0, then time steps 2 to 11 at time step 1, and so on, and finally it will forecast time steps 50 to 59 at the last time step. So each target must be a sequence of the same\n", "length as the input sequence, containing a 10-dimensional vector at each step. Let’s prepare these target sequences:" ], "metadata": { "id": "2lxUhMIvNId7" } }, { "cell_type": "code", "execution_count": 37, "metadata": { "id": "sRRkwZGd5jlA" }, "outputs": [], "source": [ "np.random.seed(42)\n", "\n", "# Notice that the model is still causal: when it makes predictions at \n", "# any time step, it can only see past time steps.\n", "\n", "n_steps = 50\n", "series = generate_time_series(10000, n_steps + 10)\n", "X_train = series[:7000, :n_steps]\n", "X_valid = series[7000:9000, :n_steps]\n", "X_test = series[9000:, :n_steps]\n", "Y = np.empty((10000, n_steps, 10)) # each target is a sequence of 10D vectors\n", "for step_ahead in range(1, 10 + 1):\n", " Y[:,:, step_ahead - 1] = series[:, step_ahead:step_ahead + n_steps, 0]\n", "Y_train = Y[:7000]\n", "Y_valid = Y[7000:9000]\n", "Y_test = Y[9000:]" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "id": "G0w2fo8M5jlA", "outputId": "95d507e9-1197-4ccb-d732-ae536a6f508d", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "((7000, 50, 1), (7000, 50, 10), (10000, 60, 1))" ] }, "metadata": {}, "execution_count": 41 } ], "source": [ "X_train.shape, Y_train.shape, series.shape" ] }, { "cell_type": "markdown", "source": [ "To turn the model into a sequence-to-sequence model, we must set `return_sequences=True` in all recurrent layers (even the last one), and we must **apply the output Dense layer at every time step**. Keras offers a\n", "`TimeDistributed` layer for this very purpose: it wraps any layer (e.g., a Dense layer) and applies it at every time step of its input sequence. It does this efficiently, by reshaping the inputs so that each time step is treated as a\n", "separate instance (i.e., it reshapes the inputs from `[batch size, time steps, input dimensions]` to `[batch size×time steps, input dimensions]`. In this example, the number of input dimensions is 20 because the previous SimpleRNN\n", "layer has 20 units), then it runs the Dense layer, and finally it reshapes the outputs back to sequences (i.e., it reshapes the outputs from `[batch size × time steps, output dimensions]` to `[batch size, time steps, output dimensions]`; in this example the number of output dimensions is 10, since the Dense layer has 10 units). Here is the updated model:" ], "metadata": { "id": "QSdLOYW3PFgj" } }, { "cell_type": "code", "execution_count": 39, "metadata": { "id": "lkFg1r3-5jlA", "outputId": "425807e6-1b3c-45b2-e8a9-09f18f961983", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_5\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " simple_rnn_8 (SimpleRNN) (None, None, 20) 440 \n", " \n", " simple_rnn_9 (SimpleRNN) (None, None, 20) 820 \n", " \n", " time_distributed (TimeDistr (None, None, 10) 210 \n", " ibuted) \n", " \n", "=================================================================\n", "Total params: 1,470\n", "Trainable params: 1,470\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n", " keras.layers.SimpleRNN(20, return_sequences=True),\n", " keras.layers.TimeDistributed(keras.layers.Dense(10))\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "source": [ "It makes it clear that the Dense layer is applied independently at each time step and that the model will output a sequence, not just a single vector.\n", "\n", "All outputs are needed during training, but only the output at the last time step is useful for predictions and for evaluation. So although we will rely on the MSE over all the outputs for training, we will use a custom metric for\n", "evaluation, to only compute the MSE over the output at the last time step:" ], "metadata": { "id": "835XgNffQAVq" } }, { "cell_type": "code", "source": [ "def last_time_step_mse(Y_true, Y_pred):\n", " return keras.metrics.mean_squared_error(Y_true[:, -1], Y_pred[:, -1])\n", "\n", "model.compile(loss=\"mse\", optimizer=keras.optimizers.Adam(learning_rate=0.01), metrics=[last_time_step_mse])\n", "history = model.fit(X_train, Y_train, epochs=20,\n", " validation_data=(X_valid, Y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ZfljBHwIPzVS", "outputId": "1b640b93-6771-4644-d48d-c7e1f76f2ce5" }, "execution_count": 40, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 27s 115ms/step - loss: 0.0508 - last_time_step_mse: 0.0400 - val_loss: 0.0429 - val_last_time_step_mse: 0.0324\n", "Epoch 2/20\n", "219/219 [==============================] - 24s 112ms/step - loss: 0.0395 - last_time_step_mse: 0.0283 - val_loss: 0.0352 - val_last_time_step_mse: 0.0244\n", "Epoch 3/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0327 - last_time_step_mse: 0.0215 - val_loss: 0.0314 - val_last_time_step_mse: 0.0208\n", "Epoch 4/20\n", "219/219 [==============================] - 26s 118ms/step - loss: 0.0295 - last_time_step_mse: 0.0181 - val_loss: 0.0275 - val_last_time_step_mse: 0.0157\n", "Epoch 5/20\n", "219/219 [==============================] - 24s 112ms/step - loss: 0.0272 - last_time_step_mse: 0.0152 - val_loss: 0.0286 - val_last_time_step_mse: 0.0203\n", "Epoch 6/20\n", "219/219 [==============================] - 24s 111ms/step - loss: 0.0249 - last_time_step_mse: 0.0123 - val_loss: 0.0230 - val_last_time_step_mse: 0.0094\n", "Epoch 7/20\n", "219/219 [==============================] - 24s 112ms/step - loss: 0.0228 - last_time_step_mse: 0.0100 - val_loss: 0.0220 - val_last_time_step_mse: 0.0086\n", "Epoch 8/20\n", "219/219 [==============================] - 24s 111ms/step - loss: 0.0215 - last_time_step_mse: 0.0085 - val_loss: 0.0221 - val_last_time_step_mse: 0.0098\n", "Epoch 9/20\n", "219/219 [==============================] - 24s 111ms/step - loss: 0.0212 - last_time_step_mse: 0.0085 - val_loss: 0.0206 - val_last_time_step_mse: 0.0073\n", "Epoch 10/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0210 - last_time_step_mse: 0.0084 - val_loss: 0.0197 - val_last_time_step_mse: 0.0071\n", "Epoch 11/20\n", "219/219 [==============================] - 24s 110ms/step - loss: 0.0205 - last_time_step_mse: 0.0081 - val_loss: 0.0200 - val_last_time_step_mse: 0.0076\n", "Epoch 12/20\n", "219/219 [==============================] - 24s 111ms/step - loss: 0.0203 - last_time_step_mse: 0.0083 - val_loss: 0.0202 - val_last_time_step_mse: 0.0086\n", "Epoch 13/20\n", "219/219 [==============================] - 25s 112ms/step - loss: 0.0197 - last_time_step_mse: 0.0074 - val_loss: 0.0208 - val_last_time_step_mse: 0.0082\n", "Epoch 14/20\n", "219/219 [==============================] - 25s 114ms/step - loss: 0.0192 - last_time_step_mse: 0.0070 - val_loss: 0.0192 - val_last_time_step_mse: 0.0070\n", "Epoch 15/20\n", "219/219 [==============================] - 25s 114ms/step - loss: 0.0191 - last_time_step_mse: 0.0071 - val_loss: 0.0185 - val_last_time_step_mse: 0.0071\n", "Epoch 16/20\n", "219/219 [==============================] - 24s 112ms/step - loss: 0.0189 - last_time_step_mse: 0.0069 - val_loss: 0.0191 - val_last_time_step_mse: 0.0085\n", "Epoch 17/20\n", "219/219 [==============================] - 24s 110ms/step - loss: 0.0188 - last_time_step_mse: 0.0069 - val_loss: 0.0185 - val_last_time_step_mse: 0.0070\n", "Epoch 18/20\n", "219/219 [==============================] - 24s 110ms/step - loss: 0.0186 - last_time_step_mse: 0.0069 - val_loss: 0.0181 - val_last_time_step_mse: 0.0066\n", "Epoch 19/20\n", "219/219 [==============================] - 24s 112ms/step - loss: 0.0185 - last_time_step_mse: 0.0069 - val_loss: 0.0176 - val_last_time_step_mse: 0.0062\n", "Epoch 20/20\n", "219/219 [==============================] - 24s 109ms/step - loss: 0.0185 - last_time_step_mse: 0.0070 - val_loss: 0.0192 - val_last_time_step_mse: 0.0077\n" ] } ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "id": "LT0dimq-5jlA" }, "outputs": [], "source": [ "np.random.seed(43)\n", "\n", "series = generate_time_series(1, 50 + 10)\n", "X_new, Y_new = series[:, :50, :], series[:, 50:, :]\n", "Y_pred = model.predict(X_new)[:, -1][..., np.newaxis]" ] }, { "cell_type": "code", "source": [ "model.evaluate(X_valid, Y_valid)" ], "metadata": { "id": "7-GHQBIcZR31" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "execution_count": 43, "metadata": { "id": "DICzAf4-5jlB", "outputId": "7b867ee3-336f-4d78-9ac7-068ace0a8fc2", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEUCAYAAABpmDIHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xW9fn4/9d1Z0IgEMIeGcjeshXBhOKqIKiYooitolSt9WOrbW3VVv06Krb92UFBhDpZ0aIgziIJS8IIexOy2JDBSELmff3+uO9AiAlk3OPc4f18PO5HknPe55z3ITe57nPO+31doqoYhmEYhrfYvN0BwzAM4+pmApFhGIbhVSYQGYZhGF5lApFhGIbhVSYQGYZhGF5lApFhGIbhVSYQGYZhGF5lyUAkIk+IyCYRKRKR967Q9lciclxEzorIf0QkqMK6KBFJEJECEdkrImPc3nnDMAyjViwZiICjwCvAfy7XSERuAZ4FfgREAp2Blyo0WQBsAcKB54BPRKSVOzpsGIZh1I1YObOCiLwCdFTVn1Wzfj6Qrqp/cP78I2CeqrYVkW7ADqClqp5zrl/tXD/LIydgGIZhXJG/tztQT72BJRV+3ga0EZFw57rU8iBUYX3vqnYkItOAaQDBwcGDIiIi3NNjC7Db7dhsVr0Yrj9zfr6rIZ8bNPzz279/f5aq1vquk68HoibAmQo/l3/ftIp15es7VLUjVZ0NzAbo3r277tu3z7U9tZDExERiYmK83Q23MefnuxryuUHDPz8RyajLdr4emvOA0Ao/l39/rop15evPYRiGYViGrweiXUD/Cj/3B06oarZzXWcRaVpp/S4P9s8wDMO4AksGIhHxF5FgwA/wE5FgEanqNuIHwFQR6SUizYHngfcAVHU/sBX4k3P7O4F+wH89chKGYRhGjVgyEOEIKOdxDM2+3/n98yISISJ5IhIBoKpfA9OBBCATyAD+VGE/k4DBQC7wZ2Ciqp7y2FkYhmEYV2TJwQqq+iLwYjWrm1Rq+zfgb9XsJx2IcV3PDMMwDFez6hWRYRiGcZUwgcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK8ygcgwDMPwKhOIDMMwDK+ybCASkRYi8qmI5ItIhojcV027r5zlw8tfxSKyo8L6dBE5X2H9t547C8MwDONKLFkq3GkGUAy0AQYAX4jINlXdVbGRqt5W8WcRSQRWVNrXOFVd7sa+GoZhGHVkySsiEQkB7gZeUNU8VV0DLAWmXGG7KGAk8IG7+2gYhmG4hiUDEdANKFXV/RWWbQN6X2G7B4DVqppeafk8ETklIt+KSH8X9tNwobUpWfzlm30kZ+R6uyuGYXiQqKq3+/ADIjIS+FhV21ZY9ggwWVVjLrNdCvCKqr5XYdkIYDMgwP85Xz1U9XSlbacB0wBatWo1KD4+3mXnYzV5eXk0adLE2924REJmCe/vLgYgwAa/GxJMlzC/Ou3LiufnSg35/BryuUHDP7/Y2NhkVR1c2+2s+owoDwittCwUOFfdBiJyA9AW+KTiclVdW+HH10Xkpzhu331eqd1sYDZA9+7dNSYmpq59t7zExESscn6qynvfp/Phnt0XlpXaoah5JDExXeq0Tyudnzs05PNryOcGDf/86sqqt+b2A/4i0rXCsv7ArmraA/wUWKyqeVfYt+K4OjK8LL+olF8u2MJLn+9mYEQYQf4X345DosK82DPDMDzJkldEqpovIouBl0XkYRyj5sYD11fVXkQaAXHAnZWWRwCdgI04gu4vgZbA2sr7MDxr6dYjvPj5bnLzi/ndrT34+ajObDl0mg/XpfPZ1qMcO1Po7S4ahuEhVr0iAngcaAScBBYAj6nqLhEZKSKVr3omAKeBhErLmwIzgVzgCHArcJuqZru158ZlfbblCE8u3EpOfjEBfjaGRrfAZhMGRYbxt7gBdGvThBkJKdjt1nt+aRiG61k2EKlqjqpOUNUQVY1Q1fnO5atVtUmltgtUNVIrjbxQ1V2q2s+5j3BV/ZGqbvLkeRg/9PfvDlz4vsxuJyn14ucCm014PKYL+0/k8b89J7zRPcMwPMyygchomL5PySItKx9/m+AnEOBvY3jn8EvajO3XjogWjZmRkIIVR3UahuFaJhAZHmO3K69/tZcOzRvx4dSh/Prm7sx7eDiDIi8dmODvZ+PxmGvYfvgMqw5keam3huF+8+ZBVBTYbI6v8+ZVvayhM4HI8JhlO46x48gZnr65G9dd05JfxHb5QRAqd9fAjrRrFsyMFSke7qVhuF55cBk9+kZeD5vO8ucSmDcPpk2DjAxQdXx98EF46CGIzkjgGZ1ORoajTUMPRiYQGR5RVFrGm9/spWe7UCYM6HDF9oH+NqaN6syG9BzWp5qxJYbvujTgCN+eHkL/1+L44MEECgoubVtSAtcXJxBPHBsZAkBBATz3nBc67kEmEBkeMS8pk0M553n2th7YbDWbxjVpSAThIYG89uUeZiSkmNQ/hk967jkuCTiJxBJHPB+VxBFTaaBvDI4gFEc8icReWJ6Z6aneeocJRIbbnS0s4Z8rDnBDl5aM6tqyxts1CvTjtj7t2Hb4DH/9dh+T5ySZYGT4nKqCSHkwiudiMKouCAFERHiip95jApHhdrMSD5JbUMKzt/VApHZJLcKbBABgVyguvXSot2H4guqCyI7wWB4IXMQSxjOTR4knjvv84vk+8NIg5OcHr77qgY56kQlEhlv9b9cJZq9KZWTXlvTp0KzW24/q1vpC6h9V6Nex9vswDG969VVo3PjSZY0bw9/fUt689TtCOcejvE1KUC9+9n4s//kPREaCCDRtCmVlcO21QEICTJ/ulXNwNxOIDLdJzsjl0XnJlNqVDWk5dbqtNigyjPmPDOeewR2xCby9MpWSMrsbemsYLjZ9OiQkMHkyzJ5dHlyUyEiY/bYyec/z9Fn6Gvj7Q8uWXFe0isnzb2fyfUp6OtjtkJrqCEaLHk2AuDgYMsTbZ+UWJhAZbrPuYBZlzjQ9pWV1v602KDKMNyf25/W7+7EmJYvfL95hJroa1jdkiCN4OINRejqsWLGS9DRHEOK11xzt/vxnOHTIcdnz5ZcwdCgUFQHQsiX8864Enlgdx96X4yE2tvrj+TBLJj01GoY2ocEA2KrJoFBbcYM7cST3PH//7gA2gcjwEIZ3Dq92LpJheFVsLMTHO4JRvDOIqMLzFYLQX/4CTz/t+D45GW67Db75Bnr0gA0bYOdOpiyL467QePIXx/K/x7x3Ou5kApHhNucKSwGYNqozN/Vq65KA8dSYrmw7dJr4TYcRICjAVmV2BsOwhIrBaNEiov/zn4uzUysGIXA8FPr6a3jgAfjwQ+jcGYKCsH38MaO2xPL009CmDZw65RgA8eqrMHmyd07L1cytOcNtNqbn0DGsEc/e1tNlgUJEGOjclwIlZiSdYXXlwWjcOCLnzXM8E6ochCr64AO4+WbIy4M+fSA2lhYtHHHq5MmLWRgaUsYFE4gMt1B1DFAYGt3C5fse0aUlgX6OYeB+tvrf8jMMt4uNhZ/8xPH9s89WH4TAMTpu82bo2hVWrYIvv+TFFx0BqKKGlHHBBCLDLQ6eyic7v5hhbghEgyLDmPuzIQgw/tr25racYX0JCfD556RPmQKzZjl+rq5d+TOlDz90RJ+4ODpnVN2+oWRcMIHIcIsNaTkADI12z9XKyK6t6N0hlEM5BVdubBjeVCG4pD/00MVnRpWDUcUgFBsLw4bBmDEQGMgnth+mA4KGk3HBBCLDLTakZdOqaRBR4Y2v3LiOhkWHsyXzNEWlZW47hmHUS+XgApcOYCgPRlW1A8cIu9xcTo6595J0QOCYFNtQMi6YQGS4nKqy3vl8qLYpfWpjaHQLikrtbD98xm3HMIy62JSew2d/+4iSiff8MLjAD4PRxo1Vtxs1CkaMoMeez9j+u3nc1HwjAI0aOSbJmlFzbiYiLUTkUxHJF5EMEbmvmnYvikiJiORVeHWusH6AiCSLSIHz6wDPncXV6XDueY6dKWRolOufD1VUvv/y24CGYQVrUrK4Z9Y69n7+HQ//+BmSO1fzJ6c8GG3cCL/9bdWTVd98E8aNg0OH+FHXQ/wh97c8/rhjBN1ddznbNIDUP5YNRMAMoBhoA0wGZopI72raLlLVJhVeqQAiEggsAT4CwoD3gSXO5YabbEwvfz7k3kAUFhJI9zZNWW8CkWEhc1alosCsYRNZ07Hv5acXxMY6glB1hgxxDPXu1g1efx1KSxk/3jFibvlyLt7S8/HUP5YMRCISAtwNvKCqeaq6BlgKTKnlrmJwTNp9S1WLVPUfgACjXdlf41Ib0nIIDfane5umbj/W0OgWJKfnUGryzxkWUFhSxtbDp7EJ+Lkio0j5VdPx43DwICxaREwMhIbC3pnVPFfyQVbNrNANKFXV/RWWbQNurKb9OBHJAY4B/1LVmc7lvYHtemlisu3O5V9X3IGITAOmAbRq1YrExMR6n4RV5eXlufX8EncV0LmpjVWrVrrtGOWani8lv7iMD5Yl0LmZH+D+8/O2hnx+vn5uCZklnC4oYXLPQIpKlR4t/DiXto3ENMf6Op2fCM3/+Ef6//a3FD79NOvbtWNq59M8+PXP2fzmHzgrAj78bwY4Hixb7QWMBI5XWvYIkFhF215Ae8APuB5HMLrXue4FYGGl9vOAFy93/G7dumlDlpCQ4LZ9nzh7XiN/t0zfXpnitmNccrwzPzyeO8/PChry+fnyuZWUlunIN1bo+H+tUbvdXmWbep3fc8+pgurEiXq+aUuNYYV+/33dd+cOwCatw998S96aA/KA0ErLQoFzlRuq6m5VPaqqZar6PfB3YGJt92O4xsY0R6kHd80fqqx1aDDRLUPMgAXD677YcYzMnAIej7nGPaNFX3oJwsLgk0/g0cdY4x/LkiWuP4w3WDUQ7Qf8RaRrhWX9gV012FZxPAfC2b6fXPqu6FfD/Rh1sCEtm0YBfvRuXzn+u8+w6BZsSMvBbjelIQzvsNuVfyccpFubJozp2cY9B1m16kJ5iOC5M3iyb4IJRO6kqvnAYuBlEQkRkRHAeODDym1FZLyIhInDUOBJHCPlABKBMuBJEQkSkSecy1e4/SSuUuvTchgUGUaAn+feWkOjW3C2sJS9x82FruEdK/aeZN+JczwWcw02mxuuhspHx739tuPnuDhe2R9H270J7N9/+U19gSUDkdPjQCPgJLAAeExVd4nISBHJq9BuEpCC43bbB8Abqvo+gKoWAxOAB4DTwEPABOdyw8XOFJSw78Q5tw/brqz8eBvSTBZuw/NUlRmJKXQMa8S4fu1df4CKWRfuvx8GD4ZNmzjzTjzxxLHlb9XkrfMhlg1EqpqjqhNUNURVI1R1vnP5alVtUqHdvaoaro75Qz3UMUS74n62qOogVW2kqgNVdYunz+VqsSkjB1X3zx+qrGNYYzo0b8SGdPOcyPC8pNQctmSe5uc3XoO/q+8EVJX6Z9Ik2LSJtoM78vtr4rllbhV563yMZQOR4Xs2pOUQ6GdjQKfmHj92+XMiNSXEDQ/7d2IKLZsEcc+gjq7feVWpf8rLSSxaRIfJsdxdFk9ewkbXH9uDTCAyXGZ9Wg79OzUjOMDP48ceGt2CrLxiDp7K9/ixjavXxxsPsfpAFrf1aeue931VqX86doSRI2HhQsaPhxUayyedL5OdwQeYQGS4xNqULLYfPk2nMPdl276cYc7Z62YYt+EpyRm5/G7xdgA+3nSI5Ixczx180iTYtYtr/XcQFgaPPw42G0RF+WbVVhOIjHpLzsjlwXc3YldYtv2oZ/9DOkWFN6ZV0yDWmwELhod8t+cE5TMGSso8XLJ+4kTw82PXCws5dw7On/ftEuImEBn1lpSaTbEz11uZXT37H9JJRBga3YL1qeY5keEZ5wpLAbC5IqdcbbVuDT/6EU2/XEhp6aXvd18sIW4CkVFvfZyTVwUv/IesYHh0C46fLSR+X7FXrsqMq4fdrqw+cIqe7Zry9M3dmffwcM+XrJ80icjSVIbww4EKvlZC3AQio972n3BM63rguijv/Id0ahIUAMDX6aVMnpNkgpHhNkmp2aRnF/DzUdfwi9gu3nnP33knRQQyiYU/WOVrJcRNIDLqxW5XPkzKYEhUGC+N7+21IARw5HQB4MjxVFLq4Xv2xlVl3oZMmjUK4NY+bb3TgenTYcsWTg66jUmyCOFiGZQLJcR9qGCeCURGvazcf4rMnAIeuC7K213humta4udMrxLg571bhEbDlpVXxLe7jnP3wI5emaoAOArhxcXR6ba+tNejTGixGoDwcGcJ8fa+VTDPBCKjXt5fl07rpkHc0ttLnwwrGBQZxh/H9gLgcW/dLjEavP8mH6akTLl3aCfvdaK8YN7MmRAUxH/vWUjLlvDjH1cIQj5UMM8EIqPO0rPySdx3ivuGRRDob4230v3DI2keJOw6esbbXTEaIFVlwYZMhkSF0dUDFYgvKzYWPv4YAFkwn5tjSyj4IgH1sSAEJhAZ9fBRUgb+NuG+odZ5MupnE4a09SNh3ynOFpZ4uztGA7POOUjhvmEWec/HxsLzz8PZs7yU+TNm5sSRMd23ghCYQGTUUUFxKfGbDnFrn7a0Dg32dncuMaydP8Wldv6364S3u2I0MAs2HKJZowBu69PO21256Omnwd+fLuvnM5PHWHLWt4IQmEBk1NGSrUc5W1jKT6+P8nZXfuCaZjY6NG/E59uPersrRgOSnVfENzuPc9fADt4bpFCVpCRHfp9mzXjCNpMTC30vE7cJREatqSrvf59Oz3ahDLbggAARYVz/9qw5kEVOvik9ZbjG4s1HKC6zc6+FbkVfKBPxxBNw5gzLRk3n10lxlP7Pt4KRCURGrc1bn8ne4+e4sVtLLq3Cbh3j+rej1K58vfO4t7tiNADJ6Tn8OzGFHm2b0s3bgxTKVaxV9OtfAzCw40nuIR69x7dqFJlAZNRKckYuf1yyE4D31qZbNntBr3ahdG4VwufbzO05o36SM3KZ9E4SuQUlHDyVZ433fOWCeR06wMCB9DjwOatssXx0R7xjvY8EI8sGIhFpISKfiki+iGSIyH3VtPuNiOwUkXMikiYiv6m0Pl1EzotInvP1rWfOoGFavtuLGYdrQUQY1689SWnZnDxb6O3uGD5swYYMSsocb3q7l5L6/kBVBfPGjsV/4zp+1D+LOQed84w2+kbBPMsGImAGUAy0ASYDM0WkdxXtBHgACANuBZ4QkUmV2oxzlhJvoqo3u7PTDd1x5x91r2QcrqVx/duhCl/sOObtrhg+aktmLku3HkMAPyu956sqmDd2LNjtPNLpa9avhzMDYx3tfIC/tztQFREJAe4G+qhqHrBGRJYCU4BnK7ZV1YrJlPaJyBJgBFSRCdCol4LiUlbsPcmw6BaM6taK4Z3DLZ29oEvrpvRsF8rn247y4Ihob3fH8DEHT+Xx0HsbadMsiD+N682+4+es/Z4fNAjatuXGc8soK7uflSvhjju83amaESvWbhGRa4G1qtq4wrJngBtVddxlthNgM/C2qs5yLksHGuG4+tsC/EZVt1Wx7TRgGkCrVq0GxcfHu+6ELCYvL48mTZrUersVmSV8sLuYPwwLpluYhYavVlLx/JalFvPJ/hLeHNWIVo2tfAOg5ur6+/MFVjm33EI7ryQVUmxXnh/WiDYhrnnvuPv8ur/5Ji1XrqRFaRY3/TiLJ59McduxqhIbG5usqoNrvaGqWu4FjASOV1r2CJB4he1eArYBQRWWjcARiBoDvweOA80vt59u3bppQ5aQkFDrbcrK7Br7lwQd+4/VarfbXd8pF6p4fpnZ+Rr5u2U6MzHFex1ysbr8/nxFbc7t253H9V8r9uum9ByX9mHlvpM6+P99q92f+1K3Hzrt0n27/Xf36aeqoL8ZkqA9erj3UFUBNmkd/uZb9SNiHhBaaVkocK66DUTkCRzPim5X1aLy5aq6VlXPq2qBqr4OnMYR6IxaWLn/FKmn8pl6Q7Rlh2xXpVOLxnRt3YT/rEmzxmgnwyVmrzrIIx9u4i/f7HdJ7SlVZd/xczz73+389D8bOJVXjF25UHnYZ4wZA4GBxJz7nL17HfNco6KsXzrcks+IgP2Av4h0VdUDzmX9gV1VNRaRh3A8OxqlqoevsG/FMcDBqIW5a9JoExrEj/taKLVJDSRn5JKWlU+pXbnvnSTmP+K9wn2GaxQUl/KvFY5bTgoUO2tP1fb3mpyew+ItR8grKmXrodNkZBdcsr7MXrf9elWTJhztHkuXncuAv6IKGRkwbZpj9eTJXu1dtSx5RaSq+cBi4GURCRGREcB44MPKbUVkMvAacJOqplZaFyEiI0QkUESCnUO7WwJr3X8WDcfe42dZk5LFA9dFWSbLdk0lpWZjdz4HLTbF8hqEt5Yf4GxhKQF+Fz9PDo9uUat9JGfkEjc7iXnrM1my9SjNGwXwyoQ+zHlgMMEBNmuNkKulmUfG0U3305X9F5YVFMBzz3mxU1dg5b8qj+N4tnMSWAA8pqq7RGSkiORVaPcKEA5srDBXaJZzXVNgJpALHMExvPs2VTV/jWrh3TXpBAfYLJVlu6aGdw6/EDxtIj75h8W4aPfRs8xdk8a9QzuxcNp1jO7RGrvC4dPna7Wf2asOUuacEOcncHPvttw/PJIxvdow7+Hh/Prm7l4te18n06dDQgIf5dwOwO18ccnqzEwsW7XVsoFIVXNUdYKqhqhqhKrOdy5frapNKrSLVtUAvThPqImqPupct0tV+zn3Ea6qP1LVTd46J1+UlVfEp1uPcNfAjoSFBHq7O7U2KDKMeQ8Pp1ubJoSFBPjWHxbjEmV25Q+f7qB5owB+d2sPBkWG8c4Dg+nfsRn/b9luThfULK/g9wezWL77BDapem7QoMgwfuGLhRWdVVuHt0ljB30Yy7JLVv+ktXWrtlo2EBnWMC8pk+JSOw/58DycQZFh3Ds0gqy8Yg7nFlx5A8OS5q/PYOuh07wwthfNGzs+FPnZhNfu6ktuQQlvfL33ivtIy8rnsY82E92qCe8/NNQ3r3yq46za+m5BHLttfRnFKkJxFIi8NSiBdwusWzDPqoMVDAtISs3i7VUHuTaiOV1ae39uR30Mi3Z84t2QlkPHsMZXaG1YzcmzhUz/eh83dGnJ+AHtL1nXu30zHhoRxTur07jz2o4MreZ50ZmCEqa+txGbwNyfDiYyPISRXVt5ovueExtL8JJ47hx7FwEFpdzCN+QFt2JxQBzBS6wZhMBcERnVSM7IZcrcDRQUl7HryFmfH/rcvW1TQoP92ZCW4+2uGLWUnJHLlP9s4HxpGa9M6FPl9IFf3dSNDs0b8av4rfzju/0/eL9uSMvmjn+tISMnn1n3DyIyPMRT3fe82FgCl3wMIrwZ+gofFls7CIG5IjKq8f3BrAuJHn1yGGslfjZhaHQL1ptA5FOSM3K5d/Y6issUf5uQnV9MVMsfBpHGgf787PooXv1yD3/73wH+uSKFB0dE075ZMOnZ+XywLgO7QoCf4O93FXz+HjMG+vQhcscOXuE5HrgmFisPNboKfiNGXZQ6J/L5QnLTmhoWHU5aVr7Jxu0jVJWZKw9S7PxApHr5zNcVJ5+WlCmzV6Xy4ue7ee/7jAsZ4y2TPdvdEhIgPR2AJ/gX+9+2djkIlwciEfmHiCyrYnmoiLwoIj0rLHtKRHaIiAmIFlJaZufzbceIbNGYX9/UrcE8zC1/dmCuiqzv5LlCHnpv42VHt1U2vHM4wQE2bAJB/jbmPDCY5OfH8OHUoQT7+/bcoFopr1W0YAEaGMgK/5sZ9jdr1yZy6a05EbkGeBS4vorVg4E/4ZioWu5tHBkRfgq868q+GHW3dNtRUrPymXX/QG7t41uZFC6nd/tQQgL9WJ+Wzbj+7a+8geFRyRm5LDtYzG5SmLM6jfyiUl66ozd92oeSlJZzxczX5UP1k1KzL2k7smsr5j3yw+UNUqWCeTJ6NMNXbeGJ8EW8H3f1jJp7CthWzVyda4EiYHf5AlU9LyIfAM9gApEllJbZ+eeKFHq2C+XmXm293R2X8vezMSiqhRmwYEHJGblMfieJwlI7HNhH55aNif/5cLq0dpTlHhRVs8wJgyLDqgw01S1vUCpXbQUYP572Xz/GxoK25P43njCLBqMa3RITkS4iUiIiL1daPtNZGXWwiAQB9wPzq9h+D/AXIAgoEREVkf86Vy8EeolIVVdRRgXJGbnMSEhx6wi2JVuPkpaVz//9qCs2W8NLyTcsugX7T+SRk1+zyY+GZySlZlNU6njGI8CEazteCEJGDVVVtXWco2rOHSxlhVq3amuNApGqpgBzgKdEJBxARP4IPATc6bwCGg40B1ZXsYsHgFTgc+A65+vXznVbcWTVvrXup9HwlX9i/Ou3++qVbbj89kdV2zuuhg7Qq10ot/RuU98uW9Iw53Mic1VkLRXn/gQF2BjRpaUXe+Ojqqra2qED9kGDudO2hFWrcKy3YNXW2gwSeBnwA54VkYdxPO+ZoqrLneuH40iGu72KbbcBHYEVqprkfGUAqKrduX54Hc/hqpCUmkVhqR27QlFJ3ZJ3OobCJvHfAyVVBrPPth4lPbuAp8Z09alSD7XRr2NzgvxtrE+7CkZO+ZAzBSUoMLydX4MZHGMVtvF3MMS+nl3fHfd2V6pV40CkqseAt4BfArOAJ1W1YhnT9sBZVa3qnkdvIBBH9dSqnHJub1SjuPRiJV0F8otKar2Pb3cdp7jMjgKFJXY+XJdeXjzwwtVQ7/ah3NSrYV4NAQT62xgYEWauiCzm/XXptA0N5uG+QSYIudr48dhQonct4/Rpb3emarUdNn0Ax3Oedao6o9K6YByDEaoyEMffz63VrD+PI9O2UYWsvCI+WJdOtzZN+NWYrgzo1JxZK1P5YvuxGu9DVdnkvAIS5+uzrUe5e+b3bEzP4a3vDpCRXcC4/u0b7NVQuWGdW7D72FnOnK99MDdcL+VkHqsPZHH/8Aj8G+BzSa/r25fzbaO4gyWstWgBnBoHIhH5EY7h1sYXOwoAACAASURBVOuAESLSr1KTbBzPiKpyLXBQVc9Ws74FkFXTvlxt/rR0F/lFZcy4byD/N6YbC5zF3Z5atIWEfSdrtI9l24+RnJHLz66P5O6uASz6+XDeuLsvR06f555Z6y4UGntr+Q/TozQ0Q6NboArJGeaqyAo+XJdOoJ+NST5YZsQniBBw1x2MYTnrlud7uzdVqumouYHApzgGLMQAmcDrlZrtBQJFpGMVu+hFhWHbVYgG9tWkL1ebb3Yd54vtx/jl6C50beMYRdQo0I+5PxtCtzZNeeyjZD5al3HZ0XRnCkp46fNd9O3QjOdv78XYawIZGh3OT4ZEkPhMLCO7XnwwXHIVFI8bGBFGoJ+N9akmEHnbucISPkk+zNh+7WjZJMjb3Wmw/O8eTyMKKfnyf97uSpWuGIhEpAvwFfAt8EvnM6CXgB+LyKgKTVc5vw6tYjengf4icouIDC8feefcf3OgW4XtDacz50t44bOd9GwXyqMx11yyLjQ4gA8eGkp4SCDPL9l52dF0r325h9yCEl6/q+8P8mw1CvTjqTHdfL4qZW0EB/jRv1Mzk2HBAhZvPkJ+cRk/vT7K211p2EaO5Hxwc3oeWEq+BS+KLhuIRKQtjgC0B5jsHOEG8AGOK6A/l7dV1XRgAzCuil39ETgBfIbj1l7PCutuB4pxXHEZFbz2xR6y84t5c2I/AqpI1BjeJOhChoDy0XRrU05d0mbdwWwWbTrEwzdE06dDsyqPUz4jvUHVZrmCodEt2HHkDPlFpd7uylXLblfeX5dO/07N6d+purv6hksEBJA7/Mf8WJeRtLbM2735gcsGIlU9rqqdVTVGVYsqLC9T1Z6qWnkS6kzgLhFpXGk/O1V1mKo2UlVR1TUVVt8PfFy5fLeItBCRT0UkX0QyROS+qvooDm+ISLbz9YZUeNouIgNEJFlECpxfB1z+n8Qa3l2TxqJNh7ijf/tqAwjATb3aEuQsha3Aok2H2ZLpuCoqLCnjuU930KlFI54a0+2yx/PZqpR1NCw6nDK78uLnuxr8MzGrWnswi9RT+fzs+khvd6Vhc5YQb/7AHbTmFOkL1v2wjZdLiLs62ehHwFHg8Zo0dgaF0Thu9VU2A8eVUhtgMjBTRHpX0W4aMAHoD/TDcUX2c+f+A4Elzn6FAe8DS5zLLWvF3hO8vMzxSO2rHccu+4dyUGQY8x8Zzm9u6c4LY3uiduXumd/z5ILNTJm7ntSsfF67sy+NAv081X2fUJ414pNNh+s1Qdiou/e/T6dlk0B+3Lfh5DO0JGcJ8catQiiRAEK+W3rp+gTvlxB3aSBS1VLgQaCm9ZjbAj9zZm64QERCgLuBF1Q1z3kFtRSYUsU+fgr8VVUPq+oR4K/Az5zrYnDk03tLVYtU9R84Ri6PrtWJedCZ8yU8+98dlM8aKim78uCB8quZqTd05ptfjWJMzzYs3XaMjem5+InQONCUnaps2yHHhArFcUvzm13WnezXEGVmF/Dd3pPcOzSCIH/zIcmtnCXEefBBUpv259pDS7DZICoKlj9XRX46L5DyCY1WIiLXAmtVtXGFZc8AN6rquEptzwA3q+p658+DgQRVbSoiv3Kuu61C+2XO9X+ttJ9pOK6uaNy48aCCgprGUteRgCBax/0/gtp1A7WD2FB7GScWPkfx0b013k/o8HtoPvJ+xOaH2ss4vfojziZ97Mae+57A9j1oM+lVxD8AELCXkbfjf5z5Ph6/puEER/SlMHNHrf7djZoLH/s0IT1v5NTiVzh/cIO3u3NViGE0S1lPU/Lpzl7ac5R47iGO/iSywlWHSVbVwbXdyKoflZsAleccnQGqyoLYxLmuYrsmzudElddVux9VnQ3MBujevbvu2+fZ0eRFpWU8/P4m1qZkMeO+gbQODb6Ytv6ve2q1r+SMXCbPSaKk1E5AUACLF77NoMiLSTASExOJiYlx8RlYR03PLzkjl6TUbLq2bsLqA1ksCgyk+cDbAEFVCfS3WXLwhq///tanZvOT2UkARNz74iX/xr5+blfizfOLioKpGYuIZxIzeZS+7CSOj0mLjEXTXXOMuk6Gt2ogygNCKy0LxZEc9UptQ4E8VVURqc1+vKa0zM4v529h9YEs3pzYj9uc98zr+gewurosxqUqlga4uXdbHou5hsfnJbP1kOOzS/mcKvPv51qLNh268L35N/aczEzI4Cek81tGk8jLPE8isUimt3tm3VLh+wF/EelaYVl/YFcVbXc511XVbhfQTy4N0/2q2Y9XbErPYfy/1vLt7hO8OK4X9wzu5JL9Xm2j4FyhffNGvDC2N+VZZq6GOVXecPKsYwDu1TJvzSoiIiCGBFrgGJjzJP8khgQiLJDQwpKBSFXzcVRyfVlEQkRkBDAe+LCK5h8AvxaRDiLSHngaeM+5LhEoA54UkSARecK53GU3ROsjOSOXSbOT2HXsLP42oW9HM5fC2wZFhvHC7b0AmHpDtAnkLlZQXEpyRi439WpzVc1bs4I5kxOIJ444FnGGUNYzlHjimDPZ+yXELRmInB7HkQj1JLAAeExVd4nISOctt3Jv46hztAPYCXzhXIYzC8QEHPWQTuOonzShmgzhHpeUmk2p3TFYRFUbfGodX/GzEVH0aNuU5btPYsXBPL7sf7tPcL6kjIdviDZX7J6UkMCY2XFs+0M8eyNvYz73MZI1bH/qP4yZHecYwu1Flg1EqpqjqhNUNURVI1R1vnP5alVtUqGdqupvVbWF8/VbrfDXQ1W3qOog52Tagaq6xRvnU5Xe7R2PrwRzi8JKRIRHRnZm34lzrDpgcvG60ufbjtKuWTBDalj623CBCiXEx7waS3o69P7rVBpzntbFRxxDt+O8G4wsG4iuBoUljoxJ9w6NMLcoLGZc//a0CQ1izupUb3elwThdUMzK/acY1799gyxDb1lVlBAf/otB7PTrR+OFcy/OM/JiCXETiLwoKTWb4AAbL97R2wQhiwn0t/HT66NYfSCLPceqq15SO8kZuZfNkt7QfbXzOCVlyh39TQ1Mj6qihHhgkLDnuqlck7OJvO+3e72EuAlEXpSUms3gyBYE+ptfgxVNHhpJowA/5qxOq/e+kjNyufedpMtmSW/olmw9QudWIRduSRveFfXcZIoIJPPFud7uiglE3pKTX8ze4+cY3tncK7eqZo0DiBvckaXbjnDibGG99vXZliMUl9qxq+OW7Bc7al5dtyE4fqaQ9Wk53HEVVAD2FYNvCWd5yAQ6Jn4ERdUV1/YME4i8ZEOaY4ScGaBgbQ/dEE2pXXn/+/Q67+PUuSK+2H7sQol2cGRXf2rhFpZuO3JV3K5btv0oqpjbclYxfTqSmED2hKmEluSQ85/PftjGgxm5rZpZocFLSs2hUYAf/czcIUuLDA/hll5tmbc+kydGd6l1AtniUju/mLeZgpJS3rynHyfOFtGrXVPWp+Xy7to0Ptt6FIAgfxvzH2m4A1aWbD1K3w7N6NyqyZUbG+7nzMgd+5eFZMyLQN+aS4vHfnJxfYWRdp5groi8JCk1m8FRYeb5kA94ZFQ0Z86X8MsFW2p95fLKF7vZkJ7DG3f3Y+KgTvwitguxPdrw7G09eHhk9IUrpKJSO8v3nHB95y0g9VQeO46cYfwAczVkGc6Rcp2emcSO8Bgi9i+HjAzHugTPZ+Q2fwW94OLzIXNbzjcIIvDdnpPcM+t7nvt0B1sycykts192JFz8pkN8sC6Dh2+IZvyADj9YP7pHG4ICbBdSCn2SfIi0LAvWca6npduOIgJj+5lAZCnOYBRzbhmgvBT1Lve2TaBwvOfLQphbc15w8fmQGajgC5JSsxEctYvsCvPWZzJvfSaNAvwoKi1DFQL8bLz74BBGdGlJckYun205woKNmVx/TTjP3tajyv1WTE7bqkkgf/56HxNnfs/7Dw29bFVeX5KcnsP769Lp1a4pbZsFe7s7RiXzjsbyIZ+wlFt4iv+P4hMzuDMonvuPxjLZg/0wgcgLyp8P9e1gng/5guGdwwn0tznKavjb+PfkgZwvtvPO6oMXMnUXl9m5f856IsIbczj3PGV2RYBHRnXG36/6Gw8VM4APjmrBlLkbmDQ7iWdu7kZ+cZlPZ09Pzsjl3jnrKS61k1foyDHnq+fSUD33HGQUx/Jf7uZeFvI1N/N1USx7noPJHoxE5tacF5jnQ76l/MqlPEnn6B5tuL1fO14Y25vgABt+AoF+wsTBHRGgzJk/UAR2H635ZNjOrZrw38euJ6xxAC9+vpu/fOPbc46SUrMpKXVkD7HbTS5FK8rMdGTkHsNyUujMzfyP8XxKZibMm+eoYVRezXXePPf1w1wReVj586FxZhirT6l45VJxWeW6T5cUJaxD/sC2zYKZcG0H/rkiBcUx6s5X6/UM7xyOCI5blyaXoiX9pHUC/zgRRxzxHKMdO+nNAu7jx/olDz4YS0mJo11GBkyb5vjeHVdK5iO5h61PNc+HGpLKdZ8qXz3VJYDEdG9NkPNqWRUG+2AQAujXsRn+NhsDI5qbXIpWlJDAuwVxPBAUTyKx7KMH/+D/CKKQxdzFiJJLk6AWFDhu5bmDCUQelpSabeYPNXD1LUo4KDKM+Y8M584B7VFgxd6Tru2gh+w9do7iMjsPjjB1nSzHOUQ7eEk898+NJTLScSv5vU5/pDi0FYfpQDxxxHBpMMp0UzVXE4g8LCk1h8FRYQRc5gG2YQyKDOP/m3Qt9w2L4O1VqaxN8b1yFMkZOUDdS94bblQhI/fkyZCeDnY7bH9iNsGPPEBfdjGLnzOESzNyR0TglowL5q+hB2XnFbHvhJk/ZNTc87f3pHOrEJ6O30ZuviXqOdbYpoxc2jULpn3zRt7uilFZFRm5AUfGhfff50ybrjwk7zGDX1xYFRTkqPJKXJyjnQuZQORBG9IcnxBNIDJqqnGgP/+YdC3Z+UX84dMdPlUxdrMZru17nJNcmxWdooMe4c+hryMCfn5wR9MEfjTbPZNdLReIRKSFiHwqIvkikiEi912m7W9EZKeInBORNBH5TaX16SJyXkTynK9v3X8G1bv4fKhhTFY0PKNPh2Y8c3N3vtp5nDe/2cuyg8WWH9J99PR5jp4pNIHIF8XGwuLFEBTEL89Px56SyuqXE5iRFceCCe7JuGC5QATMAIqBNsBkYKaI9K6mrQAPAGHArcATIjKpUptxqtrE+brZXZ2uiYS9p2gTGsT2w2e82Q3DBz0ysjN9O4Ty78RU/nugxPLzi8r7NjjSjA71SbGx8MEHUFICo0Zx3d/u4V8j43l4Xixp9S/P9QOWCkQiEgLcDbygqnmqugZYCkypqr2qTlfVzapaqqr7gCXACM/1uOYS9p4gM7eAjOwCy/8RMazHZhNGdGkJOFINlTjnF1lVckYujQL86NGuqbe7YtRVXBzccw8cOQIiTHtnCHY79OpVxSTXeg5gsNqE1m5Aqarur7BsG3DjlTYUR7WtkcDblVbNExEbsAX4japuq2b7acA0gFatWpGYmFj73l/G3zadBxx/RIpL7CxYvpFz1wS69Bg1lZeX5/Lzs5KGen4tC8ucOe8UmwhBpzNITDzs7W5VaeXO80Q2hbWrV9Vqu4b6uyvnS+fXfMsWen37Lfn9+tF8+3aCh/ciqHQXZ8scHy4yMmDq1DIKv/qAKct+xe4//anuB1NVy7xwBJLjlZY9AiTWYNuXcAStoArLRgCNgMbA74HjQPMr7atbt27qSueLS7X/S99o9LPLtPOzy7T781/qpvQclx6jNhISErx2bE9oyOc3MzFFI3+3TN9emeLtrlQrv6hEO//+C33z67213rYh/+5Ufej8VqxQbdnS8VVVdfJktYPuo4sGU6COqdaqMazQLNvFdsAmrcPffo/emhORRBHRal5rgDygckH7UODcFfb7BI5nRber6oWat6q6VlXPq2qBqr4OnMYR7Dzq0y1HOF1Qwh/H9qrXjHvDeGhENE0DYJuFnzNuO3SGMrua97ivqqoe0UcfMZ976UoKWxhAEIXEkEA8cdxjr/8ABo/emlPVmMutdz4j8heRrqp6wLm4P7DrMts8BDwLjFLVK92nUC5Wa/aIMrvy9sqD9OvYjJ9eH4XjDqJh1E2gv41h7fz53+4TnDlfQrNGAd7u0g9sznQ8/7w2wmQP8TmXKYr3XOR8/DLsTGIRKXQhiCLiiCctsv6j6Cw1WEFV84HFwMsiEiIiI4DxwIdVtReRycBrwE2qmlppXYSIjBCRQBEJdg7tbgmsde9ZXOrrncdJzy7g0RuvMUHIcIkRHfwpLrXzxfZj3u5KlTal59C1dROaN/bOM1CjHipkXKjs1VdhauOFrGMYHTnCMsayoXEsr75a/8NaKhA5PY7juc5JYAHwmKruAhCRkSKSV6HtK0A4sLHCXKFZznVNgZlALnAEx/Du21TVY0ONVJVZKw8S3TKEW3q39dRhjQYuKtRGl9ZNWLzZegMV7HZlc+Zpc1vOV1WXcQFH1u0lTyXQzXaQv/Mkd9iWseSpBJdk47baqDlUNQeYUM261UCTCj9HX2Y/u4B+Lu9gLaxNyWbHkTP8+a6++NnM1ZDhGiLCXQM7MP3rfWRk5xMZHuLtLl2QmpXHmfMlJhA1RAkJjJkdB8vj+b/YWEiYwJi4OBhT/2dEVrwiajBmrkyhddMg7hzYwdtdMRqYCQM6IAKLNx/xdlcusSnd8XzIBKIGpqpnR850QMTFOdbXgwlEbrL98GnWpmQz9YZogvz9vN0do4Fp37wR118TzuIthy2Vfy45I5cWIYFEt7TOVZpRT5cZwOCqYGQCkZvMWnmQpsH+3DcswttdMRqou67tyKGc82yyUJaO5IxcBkaEmYE5DcllBjAAF4PRxo1Vr68BE4jc4PNtR/hyx3Fu7tWGpsHWG15rNAy39mlL40A/ywxayMkvJjUr39yWa2guM4DhgthYR7s6MoHIxZIzcnlqkSOL0LLtx0xOOcNtQoL8ubVPW5ZtP0ZhSZm3u8PmDPN8yKgbE4hcbM2BU5TZHffsS8usnZjS8H13D+zIucJSfvPJNq9/6NmUkUuAn5gyJ0atmUDkas574zaBAH+bKYJnuFWgv+O/8Ofbjnk9q/vKfSdp2SSIXUfPeq0Phm8ygcjFth86TXhIIL++qZvJKWe43Ya0nAs5q7xZGmLdwSz2HD/H8TOFXg+Ihu8xgciFcvKLWbn/FBMHdeSJ0V1NEDLcbnjn8AtXRSLitSvwOWsc1dJ8oVaSYT0mELnQFzuOUWpXxg8wE1gNzxgUGcb8R4bTKawRLUICuLaT5xONFpfa2Zp5GhHwM7ekjTqwXIofX7ZkyxG6tWlCT1OV0vCgQZFhPHNLd/5v4Va+P5jNDV1bevT4izcfJju/mBfG9qSwxM7wzuHmboBRKyYQucihnAI2ZeTym1u6m8l8hsfd0rstzRoFsHBjpkcDUWmZnX8nOsqcPDQi2rz3jToxt+ZcZOm2owCMH9Deyz0xrkbBAX7ceW0Hvt11gpz8Yo8d9/PtR8nMKeCJ2C4mCBl1ZgKRC6gqn205wpCoMDqGNfZ2d4yr1KShnSgus3ss04LdrvxrRQo92jZlTM82Hjmm0TCZQOQCu4+d5cDJPO4wgxQML+rRNpQBnZqzcOMhjyRC/WrncQ6eyueJ0V2wmTInRj2YQOQCS7Yexd8m3N63nbe7Ylzl7h3aiZSTeW6fx6Oq/HPFATq3CuG2PuZ9b9SPCUT1VGZXlm49yo3dWtEixJRGNrxrbL/2hAT6sXDjIbce57s9J9l7/By/iOliij4a9Wa5QCQiLUTkUxHJF5EMEbnvMm1fFJGSCmXC80Skc4X1A0QkWUQKnF8HuLq/69OyOX62kPHXmttyhveFBPlzx4D2LNt+lLOFJW45RnJ6Ds8v2UnrpoHcYQbnGC5guUAEzACKgTbAZGCmiPS+TPtFqtqkwisVQEQCgSXAR0AY8D6wxLncZZZuPUpIoB83mYe1hkX8ZEgEhSV2lm496vJ9J2fkcu876zl+ppDcghK2Hz7j8mMYVx9LBSIRCQHuBl5Q1TxVXQMsBabUYXcxOOZJvaWqRar6D0CA0a7qb1JqFp9uOcLgqDAaBZoqrIY19O/YjB5tm7JwY6bL952Umk1xmR1wjJozqXwMV7DahNZuQKmq7q+wbBtw42W2GSciOcAx4F+qOtO5vDewXS8dPrTdufzryjsRkWnANIBWrVqRmJh42Y6m5Jbx5w2FlCqsPZDFnE+/o0uYbwSjvLy8K56fLzPnB4PCSpi3p5ip//6G69r5u+y96Z9beuF7P4Gg0xkkJrpuuLj53V2drBaImgCVc8ifAarLmRMPzAZOAMOA/4rIaVVd4NxX5fsG1e5LVWc790X37t01Jibmsh3dueIApc54qUBR80hiYrpcdhurSExM5Ern58vM+YFt3ynm7dnAd5mlrD1md1km+Eap2bAhiQkD2jPluiiXp/Ixv7urk0cDkYgkUv3VzVrgl0BopeWhwLmqNlDV3RV+/F5E/g5MBBYAebXZV205a9+ZukOGJe04evEzWHk2bFcEjVUHTuFnE16e0IfQ4IB67ctut5OVlcXp06cpK3NUmG3WrBl79uypdz+typfPz8/Pj+bNm9OyZUtsNtc+1fFoIFLVmMutdz4j8heRrqp6wLm4P7CrpoeAC+VZdgFPi4hUuD3XD8dgiHopsytLtx2lY1gjJg3pxHXXtDRJHg1LGd45nEA/G8Vldmw215WHWH0gi2s7Na93EAI4fPgwIkJUVBQBAQGICOfOnaNp04abNNhXz09VKSkp4cSJExw+fJiIiAiX7t9SgxVUNR9YDLwsIiEiMgIYD3xYVXsRGS8iYeIwFHgSx0g5gESgDHhSRIJE5Ann8hX17eey7UdJOZnH72/raeoOGZY0KDKMeY8Mo2mQP73ahbrkPZqTX8yOI2cY1a2VC3oI+fn5dOjQgcDAQJOnzuJEhMDAQDp06EB+fr7L92+pQOT0ONAIOInjFttjqroLQERGikhehbaTgBQct9s+AN5Q1fcBVLUYmAA8AJwGHgImOJfXWWmZnbeWH6BH26bc1qdtfXZlGG41JKoFD46IYvuRMxw9fb7e+1uTkoUqjHRhdm9X3+Ix3Mtdvy+rDVZAVXNwBJCq1q3GMQih/Od7r7CvLcAgV/bvs61HScvK5+0pg0x+LcPyJg7qxD9WpLB482GeGN21Xvtatf8UzRoF0K+j54vvGQ2b+ThSCyVldv7x3QF6tw/l5l5mAqthfRHhjRkW3YJPkg/XKxGqqrL6wClu6NLSpPQxXM4EolpYvPkwmTkF/PqmbuaetuEz7hncifTsAjam1z0R6v4TeZw4W8Sobp6t/mrUzosvvkifPn283Y1aM4GohopL7fzjuxT6d2rO6B6tvd0dw6ixH/dtS0igHx9vqnsi1FX7TwEwsqtrBio0BJs3b8bPz48RI0bUaruYmBieeOKJKze8iphAVEN/+XYfR06f547+7czVkOFTGgf6M7Zfe77YcYz8otIrb1CFVQdO0aV1E9o3b+Ti3tXfvHkQFQU2m+PrvHmeOe6cOXN4/PHH2blzp8/ODbIKE4hqYH1qNrNXpQLw5jf73F7rxTBc7Z7BHSkoLuOLHcdqvW1hSRkb0nIYZcGroXnzYNo0yMgAVcfXadPcH4zOnz/P/PnzmTZtGhMnTmTu3LmXrE9KSmL06NGEhITQrFkzRo8ezdGjR3n00UdZuXIlM2bMQEQQEdLT00lMTEREyMrKurCP9PR0RIRNmzYBUFZWxtSpU4mOjqZRo0Z07dqV6dOnY7fb3XuyHmC5UXNW9N736Re+d+UsdcPwlEGRYXRuGcInmw4TN7hTrbbdkJZDUamdkR54PvS73wWxe/eV25VLSoKiokuXFRTA1Knwzjs128eAAfDWWzU/JsAnn3xCZGQkffv2ZcqUKcTFxfH6668TEBDAtm3biI2NZcqUKfztb38jKCiIVatWUVpayhtvvEFaWho9evTgtddeAxy5LdPT0694TLvdTocOHYiPj6dVq1Zs2LCBadOmER4eztSpU2t3AhZjAtEV2O3K9sOnEUw6H8N3iQgTB3dk+tf7SM/KJ6plSI23XbX/FIF+NoZHW+99XzkIXWm5q8ydO5cpUxxFAW688UYaN27MkiVLmDhxItOnT2fAgAHMnj37QvuePXsCjswKgYGBNG7cmLZtazcPMSAggJdffvnCz1FRUWzevJkFCxaYQNTQLd9zgiOnC3lqTFcC/BxByFwNGb7o7oEdefPrfTy7eDu/uaVHjd/Hqw9kMSTaM6VO3nijiKZNa14yLCrKcTuusshIcFeS65SUFNasWcP8+fMBR5CfPHkyc+fOZeLEiWzZsoU777zTLceeNWsWc+bMISMjg/Pnz1NSUkJkZKRbjuVJJhBdhqoya+VBOrVoxBOxXfD3M4/UDN91OPc8IpCUmsPkOUk1ysh9/Ewh+06c466BPTzUy9p59VXHM6GCgovLGjd2LHeXOXPmUFZWdkm+tfI5WocO1W1kYnnGgopzvUpKLq2wu2jRIp566in+8pe/cP311xMaGsqMGTP49NNP63RMKzGB6DI2pueyOfM0L4/vbYKQ4fOSUrMp/ztXVFKzZ52rD1h72PbkyY6vzz0HmZkQEeEIQuXLXa20tJT333+f119/nbFjx16ybsqUKbz77rtce+21rFhRfUrLwMDAC9nGy7Vq5fj3PXbs2IXvt27dekmbNWvWMGzYsEuGfh88eLBe52MVJhBdxqyVB2kREsg9g2r3cNcwrGh453CCAmwUlthRIDK88RW3WXUgi5ZNgujZzroZoydPdl/gqeyLL74gKyuLRx55hPDwS5+ZTZo0iVmzZrF48WKuu+46pk2bxi9+8QuCg4NZvXo1N998M2FhYURFRbFhwwbS09Np0qQJLVq0oEuXLnTq1IkXX3yRP//5z6Snp/PKK69csv9u3brx3nvv8dVXX9GlSxcWLlzIypUrrxcJNgAAEedJREFUCQvz/UcF5mN+NfYdP8eKvSf52fVRpgy40SAMigxj3sPDeSzmGhoH+vHB9xnY7dWn/bHblcS9J2jTNIjNmac92FPrmjt3LrGxsT8IQgD33HMP6enpnDp1iuXLl7N3716GDx/OsGHDWLhwIQEBjtIZzzzzDIGBgfTq1YtWrVqRmZlJQEAACxcuJDU1lf79+/OnP/3pwqi6cj//+c+Ji4vjvvvuY8iQIaSnp/P000975LzdTeqTf6qh6t69u97+8iK+3nWc758dTfPGNX946gsaepVIc35X9vGmQ/zmk+28PL43D1wXVWWbV7/YzTur0xAgKMDmsiqv5fbs2XNhNFk5X63XU1MN4fyq+r2VE5FkVR1c232aK6IqlNphybaj3Ds0osEFIcMAmDioI6O6teLPX+3lUE7BJetUlbeW7+ed1WmOn7k4f84w3MEEoiqcLVYEmHpDtLe7YhhuISK8fldfbCI8u3j7hdFapWV2/vDpDt5afoCYbq0IDrDhZ+bPGW5mBitU4WyxcnuXlpbMq2UYrtKheSN+/+MePPfpThZtPMT4AR345YItLN9zgl/EXsMzN3dnc+ZpklKzzfw5w61MIKrGutRskjNyzX8+o0G7d0gEy7Yd4+XPd/P37w5w7EzhJc+NBkWGmf8DhttZ7taciLQQkU9FJF9EMkTkvsu0/UpE8iq8ikVkR4X16SJyvsL6b2vaj9Iyc0/caPhsNuH+4REUlJRx7EwhAX5C7/bNvN0t4ypjxSuiGUAx/397dx9cVZ0ecPz7XBISCCDykrDJusQAYQPIS1TU8iKOOECjpS2+oWWlirHhZUTrjs6ggOKAOLzEbsUKshJhh63jUlq12BkhKS8VhxexNkRRZGMNSbtaCAkBwsLTP85JuLkESG7uzbnn8nxmzpD7+51zeJ7ce/Pce87v/A6kAcOBD0Xkc1UtDV1RVScFPxaREiD0SrJ7VPXj1gZhx8TN1eL3P9YREDivzpBtm9TXtLeY+kYkIinAFOAFVa1V1Z3AvwDTWrBtJjAGeKetcVybJBEfqmpMrLo1qycdE2xQgvFOrH0jygb+qKqHgto+B25vwba/AHao6u9D2n8jIgHgM+CXqvr5lXZ0TZJYETJXjYYLXW1QgvFKrBWiLsCJkLZqoCVXgP0CeDmk7WFgPyDAk8C/icjPVfWiy8RFJB/IB2fep5JoTd0bA2pray0/H4tWfoMFao58T8mRiO+6Wddccw01NTVN2s6dO3dRWzyJh/xOnz4d+defqrbbApTgXB/X3LITGAHUhWzzt8D7V9jvaKAW6HKF9b7EOWd02Tizs7M1nhUXF3sdQlRZfv5w8ODBi9pOnDjRso2XLlXdtu3y62zb5qwXQ1qcXwxr7nlrAOzVMGpDu54jUtVxqiqXWEYDh4AEERkQtNkw4KKBCiEeATapau2VQsD5dmSM8bObb4b774fi4ub7i4ud/ptvjsp/P3369MZbfQcvoTNm+4mI8N5773nyf8fUYAVVPQlsAl4SkRQRGQVMBtZfahsR6QTcD6wLaf+ZiIwSkY4ikiwivwR6AbuiloAxpn3ccQe8+27zxaihCL37rrNelIwfP57Kysomy5AhQ1q9n/r6+ihE5y8xVYhcM4FOwP8CG4ECdYdui8gYEQn91vPnwHEg9KNRV+AN4BhQAUwEJqmqXRxkTDxorhi1UxECSEpKok+fPk2WhIQEtm/fzi233EJycjJpaWk89dRTTYrNuHHjKCgo4JlnnqF3796MGjUKgIMHD5KXl0fXrl1JTU1l6tSpVFVVNfk/i4qKuOGGG0hKSiItLY1HHnmksW/FihUMHTqUlJQUMjIymDFjBsePXzgdXl1dzbRp00hNTSU5OZmsrCwKCwsB57bj4MwgLiKNj9tLrA1WQFX/D6e4NNe3A2dAQ3DbRpyCFbpuKTA0GjEaY6Ij6dln4eDB1m2Ung4TJsBPfgKVlZCTAy++6CwtMXw4uH+Q26qiooJJkyYxbdo01q1bx+HDh5kxYwaBQIDly5c3rrdhwwby8/PZsWMHqkplZSVjx47lscceY9myZZw9e5Z58+YxefJkPvnkEwKBAG+++SZPPvkkixcvJi8vj9ra2iY34AsEAhQWFpKVlUV5eTlz5sxhzpw5rF/vHFB6/vnn+eKLL/jggw9IS0vjyJEj/OEPzo0P9+zZQ2pqKmvWrOHuu++mQ4f2vfVNzBUiY4xplWuvdYpQwy1a2+lGcR999BFdulz4XDxmzBhyc3NJT09n1apVBAIBcnJyeOWVV3jiiSdYtGhR47rXX399k8I0f/58hg0bxtKlSxvb3nnnHXr06MHevXsZOXIkixYtYu7cuTz99NON69x4442NP8+dO7fx58zMTF599VUmT55MUVERgUCA8vJycnNzGTlyJAB9+/ZtXL/hrrDdu3enT58+kfj1tIoVImNMzDizdCkdW3u/nobDcS+8AG+8AQsWRP2wHMDYsWNZvXp14+NOnToxZ84cbr31VgKBC2c9Ro8eTX19Pd988w3XX+/M6B9cQAD27dvH9u3bmxS2BocPHyYzM5OKigruvPPOS8azbds2lixZQllZGdXV1Zw7d476+nqqqqpIT0+noKCAe++9l3379nHXXXdxzz33cPvtLblEM/qsEBlj/Cv0nNAdd7TbOaLOnTvTv3//Fq8vcmHAbkpKSpO+8+fPk5eXx7Jlyy7aLi0tjVOnTl123+Xl5eTl5fH444/z0ksv0bNnT/bv38/UqVMbz09NmjSJ8vJytmzZwtatW8nLy+O+++7j7bffbnEO0RKLgxWMMebKmhuYcLnRdO0gJyeH3bt3c/78+ca2nTt30rFjR/r163fJ7XJzcyktLaVv377079+/ydIweCEjI4OtW7c2u/3evXupr69n5cqV3HbbbWRnZ3P06NGL1uvVq1fj+au1a9dSVFTEmTNnAEhMTOTcuXNt/A2ExwqRMcZ/Ljc6zsNiNHPmTI4ePcrMmTMpKyvjww8/5LnnnmP27Nl07tz5ktvNmjWL6upqHnjgAT799FO+/fZbPv74Y/Lz8xtnYpg3bx6FhYWsXLmSQ4cOceDAgcbzTAMGDOD8+fMUFhZy5MgRNm7c2DgirsH8+fPZvHkzX3/9NWVlZWzatImsrCySkpIA57zS1q1bqaqq4tixY1H6DTXPCpExxn/27Ln84beGYrRnT7uGlZGRwZYtW/jss88YPnw4jz76KFOnTmXx4sWX3S49PZ1du3YRCASYOHEigwcPZtasWSQlJTUWioKCAl5//XXWrFnDkCFDmDhxIqWlzrX+Q4cO5bXXXmPFihUMGjSIt95666LDfElJScybN49hw4YxatQoampqeP/99xv7ly9fTnFxMddddx0jRoyI8G/m8kTdWwSbCwYOHKhfffWV12FETUlJCePGjfM6jKix/PyhrKyMnJycJm01NTV0be1gBR+Jh/yae94aiMg+Vb2ptfu0b0TGGGM8ZYXIGGOMp6wQGWOM8ZQVImOMMZ6yQmSM8YwNlvKXaD1fVoiMMZ5ITEy84owBJracOnWKxMTEiO/XCpExxhOpqalUVFRQV1dn34xinKpSV1dHRUUFqampEd+/zTVnjPFEt27dADh69Chnz54F4PTp0yQnJ3sZVlT5Ob/ExETS0tIan7dIskJkjPFMt27dmvxhKykpafer+ttTvOcXLjs0Z4wxxlNWiIwxxngq5gqRiMwWkb0ickZE1rVg/adEpEpETojIr0UkKagvU0SKRaRORL4UkfFRDd4YY0yrxVwhAo4CLwO/vtKKIjIBeA64E+gLZAHBN6rfCHwG9ATmAe+JSO9IB2yMMSZ8MVeIVHWTqm4GfmzB6o8Aa1W1VFWPAYuA6QAikg3kAgtU9ZSq/g74ApgSnciNMcaEw++j5gYD/xz0+HMgTUR6un3fqmpNSP/g5nYkIvlAvvvwjIj8VxTijRW9gB+8DiKKLD//iufcIP7zGxjORn4vRF2A6qDHDT93baavoT+juR2p6mpgNYCI7A3nnhp+Yfn5WzznF8+5wdWRXzjbteuhOREpERG9xLIzjF3WAsFXVzX8XNNMX0N/DcYYY2JGuxYiVR2nqnKJZXQYuywFhgU9Hgb8j6r+6PZliUjXkP7S8DMwxhgTaTE3WEFEEkQkGegAdBCRZBG51CHEd4DHRGSQiHQHngfWAajqIeAAsMDdx18AQ4HftSCM1W3NI8ZZfv4Wz/nFc25g+TVLYm2yQRFZCCwIaX5RVReKyM+Ag8AgVf3OXf9p4FmgE06R+RtVPeP2ZeIUpluA74BZqvpx9LMwxhjTUjFXiIwxxlxdYu7QnDHGmKuLFSJjjDGeskIURER6iMg/ichJESkXkYe8jqktLjdvn4jc6c6/V+fOx9fXozDDIiJJIrLWfZ5qROSAiEwK6vd1fgAiskFEKt15FA+JyIygPt/nByAiA0TktIhsCGp7yH1eT4rIZhHp4WWM4XIvVzktIrXu8lVQn+9zFJEHRaTMzeGwiIxx21v92rRC1NTrQD2QBjwMvCEizc7E4BPNztsnIr2ATcALQA9gL/CP7R5d2yQA/w3cDlyDM2LyXXei23jID2AJkKmq3YA/A14WkRvjKD9w3nN7Gh6477c3gWk478M6YJU3oUXEbFXt4i4DIT5yFJG7gKXAX+NMIDAW+Dbc16YNVnCJSApwDBjiDv1GRNYDFar6nKfBtZGIvAz8VFWnu4/zgemq+ifu4xScaUdGqOqXngXaRiLynziT3vYkzvITkYFACfAk0J04yE9EHgT+EmckbH9V/SsRWYxTfB9y1+kHlAE9Q6brinkiUgJsUNW3Qtp9n6OI/AfOPJ9rQ9rD+tti34guyAb+2FCEXJecm87nBuPkBoCqngQO4+NcRSQN5zksJY7yE5FVIlIHfAlUAv9KHOQnIt2Al4CnQ7pCczuMc5Qiu/2ii6glIvKDiOwSkXFum69zFJEOwE1AbxH5RkS+F5G/F5FOhPnatEJ0QRfgREhbNc7XznhzqXn4fJmriCQCvwGK3E9dcZOfqs7EiXsMziGPM8RHfotwPlF/H9IeD7k1eBbn1jQZOBd6vu9++/F7jmlAInAvzutyODAC5/B4WLlZIbrgapqbLm5yFZEAsB7nE+Vstzlu8gNQ1XOquhP4KVCAz/MTkeHAeGBlM92+zi2Yqn6qqjWqekZVi4BdwJ/i/xxPuf/+SlUrVfUHYAVtyM0K0QWHgAQRGRDUFq9z0zWZo889jtsPn+UqIgKsxfmENkVVz7pdcZFfMxK4kIef8xsHZALfiUgV8AwwRUT2c3FuWUASzvvT7xQQfJ6je++373HyaWx2/w3vtamqtrgL8Fucu7qmAKNwvlIO9jquNuSTACTjjL5a7/6cAPR2c5viti0Fdnsdbxj5/QOwG+gS0u77/IBU4EGcQx0dgAnASZzRc77OD+gM9AlalgHvuXkNxjlEPsZ9H24Afut1zGHk2N19zhrecw+7z192POSIc35vj/s6vRbYgXO4NazXpucJxdKCM9xws/uC+Q54yOuY2pjPQpxPKsHLQrdvPM4J8FM4o7EyvY63lbn1dfM5jXM4oGF5OE7y6w38O3Dc/aP1BfB4UL+v8wvJdSHO6LKGxw+577+TODe+7OF1jGE+f3twDkkdx/nAdFe85IhzjmiVm1sV8HdAstvX6temDd82xhjjKTtHZIwxxlNWiIwxxnjKCpExxhhPWSEyxhjjKStExhhjPGWFyBhjjKesEBljjPGUFSJjfEJEuonIQhHJ8ToWYyLJCpEx/nETsADnqnZj4oYVImP8YwTObSAOeh2IMZFkU/wY4wMiUgb8PKR5k6pO8SIeYyLJCpExPiAiN+PMDl8KLHabK1W13LuojImMBK8DMMa0yOc4N8b7laru9joYYyLJzhEZ4w+DgY7Afq8DMSbSrBAZ4w+5OPdfOuB1IMZEmhUiY/xhBHBYVU94HYgxkWaFyBh/GIQN2zZxygYrGOMPx4FcEZkAVANfq+qPHsdkTETY8G1jfEBEhgBrgaFAMjBGVXd6G5UxkWGFyBhjjKfsHJExxhhPWSEyxhjjKStExhhjPGWFyBhjjKesEBljjPGUFSJjjDGeskJkjDHGU1aIjDHGeOr/AROPYplvrlCoAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n", "plt.show()" ] }, { "cell_type": "markdown", "source": [ "You might find https://www.tensorflow.org/api_docs/python/tf/keras/utils/timeseries_dataset_from_array or https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/TimeseriesGenerator useful" ], "metadata": { "id": "C2xRfWneMqEu" } }, { "cell_type": "markdown", "metadata": { "id": "rAKE21VC5jlB" }, "source": [ "## Deep RNNs with Layer Norm" ] }, { "cell_type": "markdown", "source": [ "Let’s use tf.keras to implement Layer Normalization within a simple memory cell. We need to define a custom memory cell. It is just like a regular layer, except its `call()` method takes two arguments: the inputs at the current time step and the hidden states from the previous time step. Note that the states argument is a list containing one or more tensors. In the case of a simple RNN cell it contains a single tensor equal to the outputs of the previous time step, but other cells may have multiple state tensors (e.g., an LSTMCell has a long-term state and a short-term state). A cell must also have a `state_size` attribute and an `output_size` attribute. In a simple RNN, both are simply equal to the number of units. The following code implements a custom memory cell which will behave like a SimpleRNNCell, except it will also apply Layer Normalization at each time step:" ], "metadata": { "id": "QnIADsRAU6lL" } }, { "cell_type": "code", "execution_count": 44, "metadata": { "id": "BQ5QfPbp5jlB" }, "outputs": [], "source": [ "from tensorflow.keras.layers import LayerNormalization" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "id": "tMtn5tDX5jlB" }, "outputs": [], "source": [ "class LNSimpleRNNCell(keras.layers.Layer):\n", " def __init__(self, units, activation=\"tanh\", **kwargs):\n", " super().__init__(**kwargs)\n", " self.state_size = units\n", " self.output_size = units\n", " self.simple_rnn_cell = keras.layers.SimpleRNNCell(units,\n", " activation=None)\n", " self.layer_norm = LayerNormalization()\n", " self.activation = keras.activations.get(activation)\n", " def get_initial_state(self, inputs=None, batch_size=None, dtype=None):\n", " if inputs is not None:\n", " batch_size = tf.shape(inputs)[0]\n", " dtype = inputs.dtype\n", " return [tf.zeros([batch_size, self.state_size], dtype=dtype)]\n", " def call(self, inputs, states):\n", " outputs, new_states = self.simple_rnn_cell(inputs, states)\n", " # in a SimpleRNNCell, the outputs are just equal to the hidden states: new_states[0] is equal to outputs, \n", " # so we can safely ignore new_states in the rest of the call() method.\n", " norm_outputs = self.activation(self.layer_norm(outputs))\n", " return norm_outputs, [norm_outputs]" ] }, { "cell_type": "markdown", "source": [ "Similarly, you could create a custom cell to apply dropout between each time step. But there’s a simpler way: all recurrent layers and all cells provided by Keras have a dropout hyperparameter and a recurrent_dropout hyperparameter: the **former defines the dropout rate to apply to the inputs** (at each time step), and the latter defines the **dropout rate for the hidden states** (also at each time step)." ], "metadata": { "id": "K7qg9UpqVdDd" } }, { "cell_type": "code", "execution_count": 47, "metadata": { "id": "jZa2bF8R5jlB", "outputId": "0f7d7af8-1726-4807-9f24-91b9c22e7e2e", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 74s 317ms/step - loss: 0.1668 - last_time_step_mse: 0.1624 - val_loss: 0.0738 - val_last_time_step_mse: 0.0686\n", "Epoch 2/20\n", "219/219 [==============================] - 56s 254ms/step - loss: 0.0639 - last_time_step_mse: 0.0567 - val_loss: 0.0569 - val_last_time_step_mse: 0.0498\n", "Epoch 3/20\n", "219/219 [==============================] - 55s 251ms/step - loss: 0.0538 - last_time_step_mse: 0.0462 - val_loss: 0.0504 - val_last_time_step_mse: 0.0428\n", "Epoch 4/20\n", "219/219 [==============================] - 74s 340ms/step - loss: 0.0472 - last_time_step_mse: 0.0386 - val_loss: 0.0444 - val_last_time_step_mse: 0.0358\n", "Epoch 5/20\n", "219/219 [==============================] - 82s 376ms/step - loss: 0.0416 - last_time_step_mse: 0.0316 - val_loss: 0.0390 - val_last_time_step_mse: 0.0284\n", "Epoch 6/20\n", "219/219 [==============================] - 54s 249ms/step - loss: 0.0384 - last_time_step_mse: 0.0274 - val_loss: 0.0367 - val_last_time_step_mse: 0.0256\n", "Epoch 7/20\n", "219/219 [==============================] - 56s 256ms/step - loss: 0.0364 - last_time_step_mse: 0.0254 - val_loss: 0.0349 - val_last_time_step_mse: 0.0239\n", "Epoch 8/20\n", "219/219 [==============================] - 65s 298ms/step - loss: 0.0346 - last_time_step_mse: 0.0233 - val_loss: 0.0341 - val_last_time_step_mse: 0.0229\n", "Epoch 9/20\n", "219/219 [==============================] - 55s 251ms/step - loss: 0.0335 - last_time_step_mse: 0.0221 - val_loss: 0.0334 - val_last_time_step_mse: 0.0227\n", "Epoch 10/20\n", "219/219 [==============================] - 69s 315ms/step - loss: 0.0323 - last_time_step_mse: 0.0207 - val_loss: 0.0313 - val_last_time_step_mse: 0.0192\n", "Epoch 11/20\n", "219/219 [==============================] - 72s 331ms/step - loss: 0.0314 - last_time_step_mse: 0.0194 - val_loss: 0.0300 - val_last_time_step_mse: 0.0180\n", "Epoch 12/20\n", "219/219 [==============================] - 57s 262ms/step - loss: 0.0311 - last_time_step_mse: 0.0192 - val_loss: 0.0301 - val_last_time_step_mse: 0.0184\n", "Epoch 13/20\n", "219/219 [==============================] - 70s 318ms/step - loss: 0.0299 - last_time_step_mse: 0.0178 - val_loss: 0.0289 - val_last_time_step_mse: 0.0168\n", "Epoch 14/20\n", "219/219 [==============================] - 56s 255ms/step - loss: 0.0295 - last_time_step_mse: 0.0173 - val_loss: 0.0288 - val_last_time_step_mse: 0.0167\n", "Epoch 15/20\n", "219/219 [==============================] - 67s 308ms/step - loss: 0.0291 - last_time_step_mse: 0.0170 - val_loss: 0.0289 - val_last_time_step_mse: 0.0167\n", "Epoch 16/20\n", "219/219 [==============================] - 55s 250ms/step - loss: 0.0288 - last_time_step_mse: 0.0168 - val_loss: 0.0280 - val_last_time_step_mse: 0.0160\n", "Epoch 17/20\n", "219/219 [==============================] - 57s 260ms/step - loss: 0.0285 - last_time_step_mse: 0.0164 - val_loss: 0.0284 - val_last_time_step_mse: 0.0164\n", "Epoch 18/20\n", "219/219 [==============================] - 56s 257ms/step - loss: 0.0280 - last_time_step_mse: 0.0158 - val_loss: 0.0275 - val_last_time_step_mse: 0.0154\n", "Epoch 19/20\n", "219/219 [==============================] - 54s 248ms/step - loss: 0.0280 - last_time_step_mse: 0.0159 - val_loss: 0.0278 - val_last_time_step_mse: 0.0150\n", "Epoch 20/20\n", "219/219 [==============================] - 55s 253ms/step - loss: 0.0276 - last_time_step_mse: 0.0154 - val_loss: 0.0276 - val_last_time_step_mse: 0.0156\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True,\n", " input_shape=[None, 1]),\n", " keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True),\n", " keras.layers.TimeDistributed(keras.layers.Dense(10))\n", "])\n", "\n", "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n", "history = model.fit(X_train, Y_train, epochs=20,\n", " validation_data=(X_valid, Y_valid))" ] }, { "cell_type": "code", "source": [ "model.evaluate(X_valid, Y_valid)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4EfwMTdzZOMf", "outputId": "86db371a-a6da-4814-b7ee-d4e7a30671ec" }, "execution_count": 49, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 2s 24ms/step - loss: 0.0276 - last_time_step_mse: 0.0156\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "[0.027597270905971527, 0.015594885684549809]" ] }, "metadata": {}, "execution_count": 49 } ] }, { "cell_type": "markdown", "metadata": { "id": "3QR4Q74s5jlC" }, "source": [ "## LSTMs" ] }, { "cell_type": "markdown", "source": [ "In Keras, you can simply use the LSTM layer instead of the SimpleRNN layer:" ], "metadata": { "id": "6VSPGUazWzI1" } }, { "cell_type": "code", "execution_count": 50, "metadata": { "scrolled": true, "id": "hFClcQIw5jlC", "outputId": "9286bdac-5599-4c79-e1c7-7db3b147024e", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_7\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " lstm (LSTM) (None, None, 20) 1760 \n", " \n", " lstm_1 (LSTM) (None, None, 20) 3280 \n", " \n", " time_distributed_2 (TimeDis (None, None, 10) 210 \n", " tributed) \n", " \n", "=================================================================\n", "Total params: 5,250\n", "Trainable params: 5,250\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]),\n", " keras.layers.LSTM(20, return_sequences=True),\n", " keras.layers.TimeDistributed(keras.layers.Dense(10))\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "code", "source": [ "# keras.layers.RNN(keras.layers.LSTMCell(20), return_sequences=True, input_shape=[None, 1]) also works\n", "# However, the LSTM layer uses an optimized implementation when running on a GPU\n", "# RNN layer is mostly useful when you define custom cells, as we did earl" ], "metadata": { "id": "-YL5WRMMXxvr" }, "execution_count": 48, "outputs": [] }, { "cell_type": "code", "source": [ "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n", "history = model.fit(X_train, Y_train, epochs=20,\n", " validation_data=(X_valid, Y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LdL-BteUXio0", "outputId": "e9748267-a240-43d3-c18b-a0c67924364d" }, "execution_count": 51, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 10s 27ms/step - loss: 0.0760 - last_time_step_mse: 0.0615 - val_loss: 0.0554 - val_last_time_step_mse: 0.0364\n", "Epoch 2/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0480 - last_time_step_mse: 0.0283 - val_loss: 0.0427 - val_last_time_step_mse: 0.0222\n", "Epoch 3/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0391 - last_time_step_mse: 0.0181 - val_loss: 0.0367 - val_last_time_step_mse: 0.0157\n", "Epoch 4/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0350 - last_time_step_mse: 0.0151 - val_loss: 0.0334 - val_last_time_step_mse: 0.0132\n", "Epoch 5/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0325 - last_time_step_mse: 0.0133 - val_loss: 0.0314 - val_last_time_step_mse: 0.0121\n", "Epoch 6/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0308 - last_time_step_mse: 0.0122 - val_loss: 0.0298 - val_last_time_step_mse: 0.0112\n", "Epoch 7/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0297 - last_time_step_mse: 0.0118 - val_loss: 0.0291 - val_last_time_step_mse: 0.0120\n", "Epoch 8/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0286 - last_time_step_mse: 0.0109 - val_loss: 0.0278 - val_last_time_step_mse: 0.0099\n", "Epoch 9/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0280 - last_time_step_mse: 0.0108 - val_loss: 0.0278 - val_last_time_step_mse: 0.0113\n", "Epoch 10/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0273 - last_time_step_mse: 0.0105 - val_loss: 0.0268 - val_last_time_step_mse: 0.0101\n", "Epoch 11/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0269 - last_time_step_mse: 0.0102 - val_loss: 0.0263 - val_last_time_step_mse: 0.0096\n", "Epoch 12/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0264 - last_time_step_mse: 0.0101 - val_loss: 0.0263 - val_last_time_step_mse: 0.0105\n", "Epoch 13/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0259 - last_time_step_mse: 0.0097 - val_loss: 0.0257 - val_last_time_step_mse: 0.0100\n", "Epoch 14/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0257 - last_time_step_mse: 0.0096 - val_loss: 0.0252 - val_last_time_step_mse: 0.0091\n", "Epoch 15/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0253 - last_time_step_mse: 0.0095 - val_loss: 0.0251 - val_last_time_step_mse: 0.0092\n", "Epoch 16/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0251 - last_time_step_mse: 0.0095 - val_loss: 0.0248 - val_last_time_step_mse: 0.0089\n", "Epoch 17/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0248 - last_time_step_mse: 0.0094 - val_loss: 0.0248 - val_last_time_step_mse: 0.0098\n", "Epoch 18/20\n", "219/219 [==============================] - 5s 25ms/step - loss: 0.0245 - last_time_step_mse: 0.0093 - val_loss: 0.0246 - val_last_time_step_mse: 0.0091\n", "Epoch 19/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0242 - last_time_step_mse: 0.0091 - val_loss: 0.0238 - val_last_time_step_mse: 0.0085\n", "Epoch 20/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0239 - last_time_step_mse: 0.0089 - val_loss: 0.0238 - val_last_time_step_mse: 0.0086\n" ] } ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "id": "H-BCnT5d5jlC", "outputId": "b2a3e1da-15b6-4093-dec0-5d20c4f99f66", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 1s 10ms/step - loss: 0.0238 - last_time_step_mse: 0.0086\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "[0.023788688704371452, 0.008560807444155216]" ] }, "metadata": {}, "execution_count": 52 } ], "source": [ "model.evaluate(X_valid, Y_valid)" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "id": "dHvxuwIw5jlC", "outputId": "8e240048-c556-4fed-84a9-e7a1eff224af", "colab": { "base_uri": "https://localhost:8080/", "height": 291 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gU1frA8e+7aQRCEmro0kIngNSoSDCAFVBAr4IKiiAqKlcRe0N/IgpX0YsKSlFBQS9wxQpSQhOleLEggiCg0jsJJW3f3x+zhCUukITsboD38zzz7O6ZszPvJJt9c2bOnCOqijHGGFPYXMEOwBhjzLnJEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLwKaYESktIjMEJFDIrJZRHqepJ6IyHAR2eNZhouIeK1XzzbSPMs7gTsKY4wxeREa4P2NBjKAOKAp8LmI/KCqq3PV6w9cCzQBFPga2Ai85VWniaqu93/IxhhjCiJgLRgRKQF0B55U1TRVXQzMBG7xUb03MFJV/1LVLcBIoE+gYjXGGHPmAtmCqQNkqeo6r7IfgHY+6jb0rPOu1zBXnYUi4gK+AR5Q1U2+dioi/XFaRERGRjavWrXq3+qE/L6VkKxMMupccNqDcLvduFxF49JVUYoFilY8FotvRSkWKFrxWCy+rVu3breqlivQm1U1IAvQFtieq6wfkOKjbjZQz+t1PM6pMvG8vhQIB2KBfwM/A6Gni6F58+bqy+web6mC7lv4o8/13ubPn3/aOoFSlGJRLVrxWCy+FaVYVItWPBaLb8AKLeD3fiBTZBoQnassGkjNQ91oIM1zsKjqQlXNUNX9wP1ADaB+QQPTzl1xI6S+N72gmzDGGJNLIBPMOiBUROK9ypoAuS/w4ylrkod6xyggp1h/StVaVWAJF1NiliUYY4wpLAFLMKp6CJgODBWREiJyMdAVeN9H9feAB0SksohUAh4EJgKISEMRaSoiISIShdMBYAuwpqCx1awJM6Qbpf/8EdZbxzRjjCkMgb6KdDcQCewEPgTuUtXVItJWRNK86o0BPgV+wrm+8rmnDJwuzlOBg8DvQHXgGlXNLGhQ4eGwvPJ1zosZMwq6GWOMMV4Ceh+Mqu7Fub8ld/kiIMrrtQJDPEvuuvOAuoUdW/EG1fl1TzPqzZgBDz1U2Js3xpjzTqBvtCyySpaEKRndeGbpk7B1K1SqFOyQjPmbgwcPsnPnTjIzC9Zgj4mJYc2aAp9NLnRFKZ7zNZYSJUpQpUoVv3SLtgQDLF0KM2fCL9ndeIYn+f1f/6XmiLuDHZYxJzh48CA7duygcuXKREZG4jV6Up6lpqZSsmRJP0RXMEUpnvMxFrfbzZYtW9i9ezfly5cv9O0XjTt5giwlBbKzYQ31+ZW6yH+tN5kpenbu3EnlypUpXrx4gZKLMbm5XC7i4uI4cOCAf7bvl62eZZKSICICQJjBdVTfmAJ79gQ3KGNyyczMJDIyMthhmHNMWFgYWVlZftm2JRggMRHmzoXbb4fpdEPc2fDZZ8EOy5i/sZaLKWz+/ExZgvFITIR33oGMxi3YHloFnW6nyYwx5kxYgvEiAoMfEqZmdcP95SxISzv9m4wxxvhkCSaXf/wDFpbpRkhmOnz5ZbDDMcb40KdPH6655pp8vScpKYmBAwf6KaLjnnnmGRo1auT3/ZwNrJtyLuHh0GbwJex6tCyh46ZT6vrrgx2SMWet053f79mzJ5MnT873dkeNGnVspPU8mz59OmFhYfnelyk4SzA+9BsQwidPduWGeR9BevqxLmbGmHzatm1bzvPPPvuMfv36nVCWu/dSZmZmnpJATExMvmMpXbp0vt9jzoydIvMhNhbSr+5GZGYqu6fODXY4xhSqpUth2DDn0d8qVKiQs8TGxp5QdvToUapWrcqHH37IZZddRmRkJGPGjGHPnj3cdNNNVKlShcjISBo2bMiECRNO2G7uU2RJSUncfffdPPbYY5QtW5by5cszePBg3G73CXW8T5FVr16d559/njvvvJPo6Gjq1avHyy+/fMJ+1q1bR7t27ShWrBh169bliy++ICoqiokTJ+b5Z+B2u3nuueeoWrUqERERNG7cmE8++eSEOkOHDuWCCy4gIiKCChUq0L9//5x1CxcupE2bNkRFRRETE0OrVq34+eef87z/YLIWzEl0Gp7MwU9K8vuI6ZS99apgh2PM3wwaBKtW5e89+/ZF8vPP4HaDywUJCZCfxkDTpvDqq/nb5+k8+uijjBgxgnHjxhEWFsbRo0e58MILefjhh4mOjmbOnDnceeedVKtWjeTk5JNuZ/Lkydx///188803rFq1ip49e9K8eXNuuummk77nlVde4dlnn+Whhx5ixowZDBkyhEsuuYTExETcbjfXXXcdFSpU4Ntvv+XIkSMMGjSI9PT0fB3fqFGjePnll3nrrbdo0aIFkyZNolu3bqxcuZKmTZsybdo0RowYwYcffkjjxo3ZuXMnKSkpgNPC69q1K3379mXy5MlkZmby/fffExISkq8YgsUSzElUrxvBkmrXUPfnTzi49y2iS9uPypz9DhwQjv1T73bDgQP5SzD+cO+999KjR48Tyh7yGnC2f//+zJs3jw8//PCUCaZBgwYMHToUgDp16vD2228zd+7cUyaYTp065bRqBgwYwNixY5k7dy6JiYl8/fXXrF27ltmzZ1O5cmXASUgXX3xxvo5vxIgRDB48mJ49ewJOa2XhwoWMGDGCSZMmsXnzZipWrEinTp0ICwujWrVq1K3rjOd78OBB9u/fT+fOnalVqxYA9erVy9f+g8m+NU+hwl3XUfbRD/n4ycVcPzop2OEYc4KCtCTmzDlKly4lyMhwOrRMnuzcAxZMLVq0OOF1dnY2L774IlOnTmXLli2kp6eTkZFBUlLSKbeTkJBwwutKlSqxc+fOAr/n119/pVKlSjnJBaBly5b5GhTy4MGDbN269W9J6ZJLLuGLL74A4Prrr2fUqFHUqFGDyy+/nCuuuIL27dtTsmRJSpcuTZ8+fbj88stJTk4mOTmZHj16UK1atTzHEEx2DeYUag28knSJIPW9GRRw8FpjipTWrd3MnQvPPeeMXhHs5ALOaL7eRowYwciRI3nooYeYO3cuq1at4tprryUjI+OU28ndOUBETrgGU1jvKSzHethVrVqVtWvXMmbMGKKjo3nwwQe59NJLOXToEAATJkzgu+++49JLL2XmzJnUrVuXWbNmBSTGM2UJ5lSiotjX8nI6pk3nPx/nr0ukMUVVYiI8+mjRSC6+LF68mM6dO3PLLbfQtGlTatWqxbp16wIeR7169di6dStbt27NKVuxYkW+ElB0dDSVKlViyZIlJ5QvXryYBg0a5LwuVqwYV199Na+88grLly9nzZo1J7ynSZMmPPzww6SkpJCUlMS77757BkcWOHaK7DTKD+iGa9lMvhi6ghtvahnscIw559WpU4epU6eyePFiypYty+uvv87GjRtp1qxZQOPo2LEjdevWpXfv3owYMYIjR47wwAMPEBoamq/xux566CGeeuop4uPjad68OZMmTWLRokV8//33AEycOJGsrCxat25NVFQUU6dOJSwsjPj4eDZu3MiYMWPo0qULlStX5vfff+fHH3/krrvu8tdhFypLMKfh6toZtyuEBmuns2CBJRhj/O2JJ55g48aNXHnllURGRtKnTx969erFL7/8EtA4XC4XM2bM4I477qBVq1ZUr16dkSNH0q1bN4oVK5bn7dx3332kpqYyZMgQduzYQd26dZk2bRpNmjQBIDY2luHDhzN48GAyMzNp0KABkyZNokaNGuzYsYN169Zx/fXXs3v3buLi4ujVqxcPP/ywvw67cKnqebM0b95cCyKrfbL+FlJHr7narfPnzy/QNvyhKMWiWrTiORdj+eWXX854GwcPHiyESApPUYonL7GsWrVKAV2xYkXQYylMp/psASu0gN+51oLJg5Ae3ag9/x42fL6GzdcXD3Y4xpgAmTFjBiVKlCA+Pp5NmzbxwAMP0KRJEy688MJgh3ZWsIv8eXHttQDcEDqdjz+uEuRgjDGBkpqaysCBA2nQoAG9evWifv36zJo1y+blySNrweRFpUqQmMjtv03nhdmPsWMHxMUFOyhjjL/deuut3HrrrcEO46xlLZi86taNarv/R5XMTYweHexgjDGm6LMEk1fXXQfApNg7WTZqKYcPBzkeY4wp4izB5NXOnSBC4v45TD+YzFdPB2AoWmOMOYtZgskrz+imAkSQzsYJKWRnBzUiY4wp0izB5FVSEhQrhgIu3MzZ04RPPw12UMYYU3RZgsmrxESYO5ct3bpBSAj3FR/PyBE2PpkxxpyMJZj8SExk/b33Is8/z5WHp1FlyRS++y7YQRlz7nvmmWdo1KjRSV/7MnDgwNMO8V+QfftL7lk6zwWWYApi8GCyW7bhDbmH8c9vPX19Y85TXbp0OekkYWvWrCE6OprZs2fne7uDBw9mwYIFZxreCTZv3oyIsGLFCr/v63xhCaYgQkMJeX8iUSFH6PJZf37fYKfKjPGlb9++zJ8/n02bNv1t3bhx46hWrRodOnTI93ajoqIoU6ZMIURYtPZ1rrEEU1B163LoiRe5ms/59s4JwY7GmLxbuhSGDXMe/ezqq68mLi6OCRNO/BvJzMzk/fff5+abb0ZV6du3LzVq1CAyMpL4+HheeumlU867kvu0VXZ2NoMHD6ZUqVKUKlWKQYMGkZ2rm+dXX31F27ZtKVWqFKVLl+byyy9nzZo1OesbN24MOLNWikjO6bXc+3K73Tz33HNUrVqViIgIGjduzCeffJKzftOmTYgI06ZNo2PHjhQvXpwGDRrw9ddf5+tnl56ezqBBg4iLi6NYsWK0adOGxYsXn/AzvO+++6hUqRIRERFUrVqVRx55JGf99OnTSUhIIDIyktKlS9OuXTt27NiRrxjOlA0VcwZin7yXX9+cwTVzB7H/h2Rim1wQ7JDM+WTQIFi1Kl9vidy3D37+GdxucLkgIQFiYvK+gaZN8zVXc2hoKL1792bixIk8/fTTOdMNf/rpp+zevZubb74Zt9tN5cqV+eijjyhXrhzLli2jf//+lClThr59++ZpPyNHjuTtt9/m7bffJiEhgdGjRzN58uQTBqU8dOgQgwYNIiEhgSNHjvD888/TuXNnfvnlF8LDw5k/fz7t27fnq6++okmTJoSHh/vc16hRo3j55Zd56623aNGiBZMmTaJbt26sXLmSpk2b5tR7/PHHefnll3njjTd4/vnnufHGG9m8eTNRUVF5OqYhQ4bw0UcfMX78eGrWrMm//vUvrrjiCn777TcqVqzIa6+9xowZM5gyZQrVq1fnr7/+Yu3atQBs376dG2+8kWHDhtG9e3fS0tL49ttv87TfwmQtmDPhcqHjJ+DCzb7ufZ0/WmOKMDlw4Pjn1O2GAwf8vs++ffvyxx9/MGfOnJyycePG0alTJ6pUqUJYWBhDhw6lZcuWVK9enRtuuIEBAwbw4Ycf5nkfr776KkOGDOGGG26gXr16jBo1igoVKpxQp3v37nTv3p34+HgSEhKYMGECGzduZNmyZQCULVsWgDJlylChQgVKly7tc18jRoxg8ODB9OzZkzp16jB06FDatm3LiBEjTqj3z3/+k86dOxMfH88LL7zA3r17WZXHfwgOHTrEm2++yfDhw7n66qupX78+b731FnFxcYz2jFW1efNm6tSpQ9u2balWrRoXXXQRt912GwBbt24lMzOTHj16UL16dRo1asQdd9xBXIAHUQxoC0ZESgPjgE7AbuBRVf3ARz0BXgTu8BS9AzzimZvAu96twLtAP1V9x5+xn0z9q2owqv5I7l8zgMx/v0XYfXcHIwxzPspHS+KYo3PmUKJLF8jIgPBwmDzZ73Mnx8fH065dO8aPH0+nTp3YunUrs2bNYsqUKTl13nrrLd555x02b97MkSNHyMzM5IIL8nZG4MCBA2zbto1Er+NwuVy0bt2aP//8M6dsw4YNPPnkk3z33Xfs2rULt9uN2+3mjz/+yPOxHDx4kK1bt3LxxRefUH7JJZfwxRdfnFCWkJCQ87xSpUoA7Ny5M0/72bhxI5mZmSfsJyQkhMTExJyJ1/r06UPHjh2pU6cOnTp14qqrruLKK6/E5XLRpEkTOnToQKNGjejUqRMdOnSgR48elCtXLs/HWhgC3YIZDWQAcUAv4E0RaeijXn/gWqAJkAB0Bu70riAipYDHgNX+DDgv6r/Sn1l0gocegvXrgx2OMSflbt0a5s6F555zHv2cXI7p27cv//3vf9m7dy8TJ06kdOnSdO3aFYCpU6cyaNAg+vTpw6xZs1i1ahV33303GRkZhRrDNddcw65duxgzZgzfffcd//vf/wgNDS20/eQewj8sLOxv6051XSm/+7nwwgvZtGkTw4YNw+1207t3bzp27Ijb7SYkJITZs2cze/ZsEhISGDduHPHx8fzwww9nvP/8CFiCEZESQHfgSVVNU9XFwEzgFh/VewMjVfUvVd0CjAT65KozDHgNpyUUVB07CSPqjeNIVhh6223YGDKmSEtMhEcfDVhyAejRowfFihVj0qRJjB8/nltvvTXnC3jx4sW0bt2agQMHcuGFF1K7dm02bNiQ523HxMRQsWLFE64xqGrOqS+APXv28Ouvv/LYY4/RoUMH6tevT2pqKllZWTl1jl1zyd05wFt0dDSVKlViyZIlJ5QvXryYBg0a5Dnm06lRowbh4eEn7Cc7O5ulS5eesJ+SJUvSo0cP3nzzTT7//HPmzZvHes8/uSJCYmIiTz/9NMuXL6dSpUpMnTq10GLMi0CeIqsDZKnqOq+yH4B2Puo29KzzrpfT0hGRVkAL4G7ghlPtVET647SIiIuLI8UzplhBpaWl+dxG865xDBz+Gu8t7s36e+/lrxtOGVahOFkswVKU4jkXY4mJiSE1NfWMtpGdnX3G2yioHj168PTTT7N//35uvPFGUlNTyc7Oplq1akycOJFp06ZRs2ZNpk2bxoIFC4iNjc2JNT09HbfbfdLXAwYM4KWXXqJq1ao0bNiQt99+m23btlG+fHlSU1MJDQ2lTJkyvPHGG5QqVYpt27bxxBNPEBoaytGjR0lNTaV06dJERkYyc+ZMypYtS0REBDExMX/b17333ssLL7xAlSpVaNq0KVOnTmXRokUsWrSI1NRU0tLSAOc6Su6f9ZEjR07688/MzCQrK4vU1FSKFStG3759GTJkCMWLF6d69eqMHj2aHTt2cOutt5Kamsq///1v4uLiSEhIIDQ0lIkTJxIdHU1MTAxz584lJSWF5ORkypcvz48//siff/5JjRo1fO7/6NGj/vl7Kehcy/ldgLbA9lxl/YAUH3WzgXper+MBxRlrMgRYAbTxrEsB7shLDM2bN8/T/NSncrL51dPTVStVdOuSsl1UIyJUC2H+9ILGEixFKZ5zMZZTzZueV4Ge693bypUrFdCLLrrohHjS09P19ttv19jYWI2JidHbb79dn332Wb3gggty6j399NPasGHDk77OzMzUQYMGaUxMjMbExOjAgQN1wIAB2q5du5w6c+fO1YYNG2pERIQ2bNhQv/rqKy1RooROmDAhJ5a3335bq1atqi6XK+e9ufeVnZ2tQ4cO1SpVqmhYWJg2atRIZ8yYkbN+48aNCujy5ctPOH5AP/7445P+fHr37q1XX311TixHjx7V+++/X8uXL6/h4eHaunVrXbRoUU79sWPHarNmzTQqKkpLliypl156qS5ZskRVnc/KFVdckfPeWrVq6fDhw0+671N9toAVWtDv/YK+Md87gmbA4VxlDwKf+qh7AGjl9bo5kOp5fi8w3mtdkUgwqqrDhqnGsU1Ti5XR1PotVTMzz3h/BY0lGIpSPOdiLGd7gvGlKMVzPsfirwQTyIv864BQEYn3KmuC74v0qz3rfNVLBq4Tke0ish24CBgpIv/2Q8z50qwZ7KACtx19k6g1y/njnuHBDskYY4ImYAlGVQ8B04GhIlJCRC4GugLv+6j+HvCAiFQWkUo4LZ2JnnV9gPpAU8+yAngWeNyvB5AH338PIvAfrmcK/6DyuGchwL02jDGmqAh0N+W7gUhgJ/AhcJeqrhaRtiKS5lVvDPAp8BPwM/C5pwxV3a+q248tON2eD6qq/+8YOw3PlDGIwD2M5kBIafTWW517Dowx5jwT0BstVXUvzv0tucsXAVFerxUY4llOt82kQgzxjHimjCElBTZuLEPvt9/m0x+74H5mKK4Xng92eMYYE1A2VEwhO3aLwdix0Pq5zkygD/rii2QvXXb6NxtzGqo2crcpXP78TFmC8aMnnoA9T7zKVq3I9it6k512JNghmbNYWFgYR47YZ8gUrszMTEJD/XMyyxKMnw1+LobFfcZR+eCvzGr5hN3kbwqsfPnybNmyhcOHD1tLxhQKt9vNjh07iMnPiNr5YMP1B8BNEzrx/S8DuGLZKzx/xbU89mVb/PQPgzmHRUdHA8dHyi2Io0ePUqxYscIM64wUpXjO11hKlCiRM5J0YbOvuQC5cO7L7K82i15z+tC/5w+M/SDKkozJt+jo6JxEUxApKSk0a9asECM6M0UpHoul8NkpskCJiiL2vxOpKRu56uPe/Kf5MLIW+X9GQWOMCRZLMIF06aXIDTfQg+lc/+MTZLdPtiRjjDlnWYIJNM9Q2yG4Cc1O5z/3zMdrxHBjjDlnWIIJtI4dITIScJJM5E/f0uemdEsyxphzjl1mDrRjt/vPnw/r1tH13Xcp+5/LuDt9Gm9Mr2AX/o0x5wz7OguGxMTjswledRWtbr6Nap+25MkrZ/Dcly0syRhjzgl2iizYbriBsGVLiC4VwlNz2vLGxZMp4C0OxhhTpFiCKQqaNiVm7XJ212rNfctu5otGQ8g8arf8G2PObpZgiopy5ai65mtWXXwPXde9zPeVr2HE4/tYar2YjTFnKUswRUlYGE0X/5spl42l2d65dHmhNQParbEkY4w5K1mCKYI2duhHB5lHDAdYlNmaabd9xsGDwY7KGGPyxxJMEZSUBCuKXUJr1wrWE89La7vwRpUXmDHdRtA1xpw9LMEUQcdulbnz+apkzlvE3stv4pHUx0nvfhP/uOYQf/0V7AiNMeb07I6LIur4rTLFIWkS2cOb8o/HHqb+F2vpVPe/DBh2AQ0bBjtKY4w5OWvBnA1ECHnkIeTzz2kUtZFvMlvwn/sXMnDghfzwQ7CDM8YY3yzBnE2uvJKQ5d8RU7MMKa7LGLm+F/c0+4YhQ+Dw4WAHZ4wxJ7IEc7apWxd5/XVcKFdlfcZCvYSsl0fSqKEya1awgzPGmOMswZyNVqwAEQBcKP9iMNO3JzLyitn0vEnZsSPI8RljDJZgzk5JSRAejtvlcob+f/hhmpTbxmwu556pbelfex7j3lHUejUbY4LIEszZyNOPedPttzv9mV98EfltHbzxBq3Kb+KTtGRq9WvPHXUW8sAD2EgAxpigsARztkpM5I9evY4P+x8RAXfdRdim9bhHvcaFUesYt74dV73SgSEXL+GVV8DtDm7IxpjziyWYc02xYrjuu5cxD23gAXmFxvzEIr2E+g9cwfXVvmPkSNi7N9hBGmPOB5ZgzlGXdIzkrWKDiHf9zqOhL9O2+EqmbWlDvcFX07niCm67DZYvD3aUxphzmSWYc9Sx4WYefb4EXRYOpsSOjTBsGJfHfMuSjJb0eL8r/Vqt4vb6S1nRYxjpKXahxhhTuGyomHOY98zMEAWPPELo3XfDa69x1YiRXH2gGdm/hqC/KhnTInjtxrlc9VwitWsHM2pjzLnCWjDnm+hoeOIJZNNGuOwyXGQTiptIjlB16ss0i0/liitg5kzItkk1jTFnwBLM+So2Fp5/HomMBJcLEeE6ncHu8Ercuqgfz3VdTs0aygsvwOefw7Bh1t3ZGJM/dorsfHbsQk1KCrRrByJEvP02N039gJ68w4Z9TfjX4/14iV4cIJawMHj3XbjxxpyBBIwx5qQC2oIRkdIiMkNEDonIZhHpeZJ6IiLDRWSPZxku4nyliUhZEVniKd8vIktF5OJAHsc5JTERHn0ULrrIeT5+PLJ1K7z5JrXqhDCagWylEhPpTavMxfTsqVSvDv36wccfW5dnY8zJBfoU2WggA4gDegFvioivWU36A9cCTYAEoDNwp2ddGnA7UA4oBQwHPhURa40VlpgYGDAAVq7kx/Er+CDkVq5jBotpy/YyDXk25l/MnbqbG26AsmWhdWt48kn46acYMjODHbwxpqgIWIIRkRJAd+BJVU1T1cXATOAWH9V7AyNV9S9V3QKMBPoAqOpRVV2rqm5AgGycRFM6AIdx3km4rTkNF73F2Ke3sv6xccTFx9DnpwfZkF6ZXR1uYsIt8wgLcTPv/5ay9b6vuDJ2KddeC2+8ARs2BDt6Y0wwiQZoREQRaQYsUdXiXmWDgXaq2jlX3QNAJ1X9zvO6BTBfVUt61fkRqAeEAe+oar+T7Lc/TouIuLi45lOmTDmj40hLSyMqKuqMtlFYghVLiY0bqfjZZ8R9/TVhqamkly1L2L59kO0m0xVB99gv+XxvEgCVKh2hRYu9lC9/lIwMF61a7aNhw4N+j9F+T74VpVigaMVjsfjWvn37laraokBvVtWALEBbYHuusn5Aio+62UA9r9fxgOJJiF7lxYCbgN55iaF58+Z6pubPn3/G2ygsQY/l8GHVSZNUa9RQhZzFXaKEHm5xia6+qK+Ob/Cydg+fqbVZpyFkqsul2rev6jffqGZm+i+0oP9svFgsJ1eU4rFYfANWaAG/9wN53SINiM5VFg2k5qFuNJDmOdgcqnoU+FBE1ojIKlW1CYQDKTISevWCmjXhssvQ9HQkNBRJTiZy3z4arP2UBjvHcZunegZhbHDX4tdx9Vgwri7vR9YjplVd6napS7vrSlNj+1KnR1tSkvcdosaYs1QgE8w6IFRE4lX1N09ZE2C1j7qrPeuWnabeMWFATcASTDAkJsK8eWwcP56at99+YnLYt4+f/rOW1+/5lVpZa6kna+lY+Ve6bv0c15FMWAAsgL0PxpLNQVwo7tBwjs6cTYkrLw3aIRljzlzAEoyqHhKR6cBQEbkDaAp0BT9hsrUAAB2oSURBVC7yUf094AER+QLn1NiDwOsAItIGJ+5lQAhwH06vtO/8fhDm5BIT+SM9nZq5Wx6lStG4Xxtua9SGlBQonwRRiUBWFmzahK75lV2L16JTpuD6YwUChGSl47qqE7PL9eBAcjeq33k5F7YtQUhI4A/LGFNwge7aezcwHtgJ7AHuUtXVItIW+FJVj13VGoPTIvnJ8/odTxlABPCaZ32mp87Vqro1MIdgCuLEcdGA0FCoXRupXZvyna+Bay+C5GQ0IwN1hbDpgo602vQVsVMmc3hKJF+EXcHGpt3Ye9E1HJBYrr3WuTfUGFN0BTTBqOpenPtbcpcvAqK8XiswxLPkrrsA55SZOZd4RhWQlBQkKYn6iYmQlcWBzxax863pXLJoOp2XzyBzeShzSWbyq924p2xXyjaMo1Yt5zKQ96NNF21M8NnNiaboyN3MCQ0l5tr2xFzbHtyjmHjPcna+NZ1uTGMsd+LePYAf/3cJ0//XjbEHr+MPLsh5a4kSl1Cnzt8TT82aUK2aMxeO9Scwxr8swZizg8tF3Vtbc/e7rXks/UWahf3M9Jun03T5dJr++E+G8k+ONGjOpgu7sVFqsHflD/yvWFc+/zmRTz+FjIxjG1JcgqeJo4SFKI8/Dt2uU+rEK2GhnqaPKixbBkuWWBYypoAswZizxvGxOYWkpMZUTWwMPA3r18OMGUROn079SY9TH6dnyM0MZySgIghe58y8T59lA0M9iw8KEBqKPPOMMwBb+fL+ODRjzklnnGBEJExVbQQqExB/6ywAULs2PPSQszz6KAwfjqg6Qz4nJSGXXHJ8+GcR/vxLePddyMoWXCFCl66CCGzfIWzbIWzfLjQ9mEInvsaFollZ8MQT8MQT7KvYgPTEdsR2acfPZdrx9U8VrIFjzEnkK8GIyH3AFlWd5nk9DugtIhuALqq61g8xGpN3XbrAqFG409NxRUTA//3f3779qwLJfY9fg2nqWe3dc+TgrCS06yLcmRlkSTgjLxjF0b/20GbbAi6Z/j7Fpr9JC6AkdVgk7VjTvh0lr2lHqcZVqFwZKld25nYz5nyW3xbMfTgjGSMilwI3AD1xBrEcCVxTqNEZk1+e82ibfN30mavaqVod0ZcnwnxnrpzwpCQeTUzE7YbNmx8hZVUWs178H8WWLaAdC+ihHxE7722YB+uphVPajhXF29Gk3FYuzpzHwiYRHG2amJN8qlRxHsuXdy71WIcDcy7Kb4KpDGz0PO8MfKyqH4nIT8CiQo3MmII62U2fBdiO9ze+ywU1akCNGqGUrdCS5OSWvJoxmGJh2aSM+oHqmxcQu2ABN6+aQd9D4+Ew6GZBUbK3DmPYV48zUa/mN+I5QGzONo8N4hYSAj16QKtWTgKqWtV5rFjRuW3ImLNNfj+2B4HywJ9AR+BlT3kmzsCTxpwXvCcDTUoKoUXihcCFwD/B7Yaff4bHH0c++wwBXGTxlD7LUzwLQHpMOfaWiWfV4Tos2h7Pb8SzLrsOX02rzdSpJU7Yl8vlJBnvpJP7eaVKeeh6vXQp1SZPhogIayqZgMhvgpkNvC0i3wO1gS895Q053rIx5rxw0tNsLhckJMBjj8HcucevB73zDhQvDr/9RsS6dVT87TfKrJ7NlUw8/t4scFeszOEq8ewtW4etxeP5PSSe1O2HiN28im//bMGifY3IPJJJGMeXcDIJ9Txf6cqkXWIm9WtnUi42k7IxmZTdu5aIMa9TIysL3nvPmfu6WzcIDw/Uj8uch/KbYO4B/g+oBvTw3JkPzr9uHxZmYMac9fJwPSgc+G5uGj//dz1t49ZRR37D9dtvRK1bR9Ty6VTbvZs2XvX/kZf9uoElniUXAeemoJtuwu0K4XD56mTXrENYozpENo5nTXYdUrbE06xLVRIvscHfzJnJV4JR1YPAvT7Kny60iIw5l+ThelDr5ChaJzfFGf81l3374PHHYcwY59SbywU33ADXXw9hYTnLz2vDGPjPMI5kOa/ffCeMytXD2LIzjD+3h5GxfBVd378eV3YG2RLKv0KHkJWp1Nm+jvjtv1Hnm4UIh2gANACOvhzB5pK1SY2L52CFOhypVofsGvFI3TqUT/uduLULcLVPonhyIiVKHO8FfsxSm3nBkP9uyg2A7GPdkUWkI870xquBl1Q1u/BDNOY8VqoU3HILTJzotDzCw+G++/72rd0oGYY1O/6lfqFndRzO6QWoDP3n8runNfVIm0T27YPNm51l0SZl3uRt7F/xG3VYRzy/0eDIOmptWEvz9V8QQc5QCDn3qWaPCuEDbmKJXMrOkrXYFVub9DKVUVcIq1Y5+TAkxAm3TRunx1xcnPNYqtTfk5I59+T3FNl44FVgrYhUBT4BUnBOnUUDjxZqdMaY3D0KCtz12rs1JUDp0s7SrBmA0Kp1JZKTK7Ekox3h4c4u6ydC+uFsdq/+g/Sf1lF84mhiF32GoISQzc3yAbfqJKf7z0HI/CucP1w1+DW7FhuoxYasWqz9Vy2+oBYbqUG6py9QWBhcEbOUdprCJ7Uj2Fs3kbi44wkoLg62bYNff4XLL3dGzraEdPbJb4KpB3zved4D+E5VrxKR9sAELMEY4x+nzR6FswtfeSyieAgRLWtAyxpQPxqS50BGBhIejsye7XRj27ABNmwgbMMGYpdtoMqCDbTVRUR7TVirIhwuVZm9pWqTShR1fp+FS7PI3vsC7/zyT1YdrsOO7FCyCSGL0JzlpxdDGeEKITI6lMioUCJLOkvx6FDiM1dTO+0HtjbswL6WlxNTJpTYWE5Y1q6Fb76B9u3tdF2g5TfBhEBOWzkZ+MLzfANOa9wYcxbLSyvIZxaqXh2SkwEoA6xbCqPnKx2a7qZlqfWwYQOyYQMlPAs/poBnhCmXZnJX6kunDswN7Pcsvqx8Fd6DvZRiN2XZTVnWUS7n+T7KMpayvF2uHCFxZdEyZXGVL0tE+RhKlRYaHlxKzP8+YdHVThfuUqXIWSIjndZTnq4r2cWnE+Q3wfwM3CUin+EkmGMtlsrA7sIMzBhTROWhNeVUEaCcs+Suv3QpJCc7XbjDw+Gjj6BpU2em06wsVq3Iot9tWbgzs4kMy+LN17NoXN9ZR3a28/jBBzBpErjdqMtFxsXtkar1KL1zN6V376b+vs1k71hJyaO7jl9D2uVZPDIJ5QDRlGYfgqILR7GeWhyhOHtxsx83IeImVNyUdbu5ATcu3Owo5iY81Fnn8tQJyUon7PABZ8MiZDdpTkjNC5BSx5tTv+8vxc9/xhLfMpb6iZ7yUqWcR69Mdq7cr5TfBPMw8F9gMPCuqh6bcbILzhTGxhhzeqfpwt00Hl6rfrwx0NjX92x0NHz8cc7puojhzxGRaztLl0LyZUp4RhoVw3YzdfRuEirthl27YPduwnbvpvTXc5AVy50u3CiVqoWxP64mGVkuMjJdpGe62LzLxe69LhTBjYtQtwvNcHE0w0U2Lty4aMoqWrEMF4pblb9W7eTQqsOUln3Esp9IPUJNnKl4+ejvh+MODcMdWYKQtAPUUHU6dgweDNddBw0bQokSJxzX2dBQym835YUiUg6IVtV9XqvGAIcLNTJjzLntNF24C3y6LneVeUJKSkmSkkqSkFjjb3VcnTsfb01FRBA1ZRxRubZ1eClcl3y8I9/cuc623W5ITYX9+yE9ZSl6ZzLuzAzcIeEsHTCFX2IS2b/f6W3+w7J0dvx2gFj2U4p9lGI/Mewn9tiStZ/LUufRkmW4AM3KQl58EV58ETfCzuja7KqQwKbYJkxcmcAqdwJDwy5g/EQXHTo4HTZCititS/ke4UhVs0XkiIg0wumxuEFVNxV6ZMYYczp5Pl13mgqnuSH2ZLnM5YKYGGehdyLUcSq5kpK46W+tqQiSk8uzN6M84eHw9ddOw2T3btizx7N8u5TsF5PRrAyyXeG81uI99qeFUmnXD1xw8EfqrvuBq5lOZ09n8YMZJfmpZ2M+pgk/kcDGkgnsKN+YYuVKcnHIUlofSWF73STSGidStiyUKcMJj+vWweLFpx5eqDJUOOUP+BTyex9MKDAMGIhzE7IA6SLyOvC4zQtjjDkr5eGG2Dx15DtFpZMlqdhYZ0ojAK5KhCuP3680OGdb1wLOoKgLPk/jqe6raZD5A03kR66q8iMtdn1AxJE3IRVIhd1/VqRUxg4EN+7vQ/j0w2vYSmX+IoRNOD31ji+hzAkNofLtIVSrEeKMrBoSAn/8AaNHU8G5xl4g+W3BvATcBAwAFnvK2uIkHRfOtRljjDE+5DVJnSzZiUDSNVEMT2lNSkprmibBBYk4mefPP+HHH+GHHyg7dSr8tA0AF9lcW2IO7vBiaFZ2TkcJzcpG3NmEkg1ZwNjCPtr8J5iewO2q+oVX2QYR2QW8gyUYY4zxu78lKhGoVs1ZrrkGLrvM6TbuuWgkX39NiK8OEMmQka4UC3czZ1Y2bVoeT0B8+y1cdx169KhSQPlNMDE497zktgE8E1wYY4wJrrx2gJgLKSlCUlIIbRJz9RC44gqYN48dF120taBh5DfB/IAzq+U9ucrv96wzxhhTFBRSB4gtsL2gIeQ3wQwBvhCRDsC3nrI2QCXgyoIGYYwx5tzjyk9lVV0I1AH+A0R5lo+By3FaNsYYYwxQsPtgtgKPe5eJSBOge2EFZYwx5uyXrxaMMcYYk1eWYIwxxviFJRhjjDF+kadrMCIy8zRVogshFmOMMeeQvF7k35OH9RvPMBZjjDHnkDwlGFW9zd+BGGOMObfYNRhjjDF+EdAEIyKlRWSGiBwSkc0i0vMk9UREhovIHs8yXETEs66OiHwiIrtEZK+IzBKRuoE8DmOMMacX6BbMaCADiAN6AW+KSEMf9frjTIDQBEgAOgN3etbFAjOBup7tLAM+8W/Yxhhj8itgCUZESuDc7f+kqqap6mKcRHGLj+q9gZGq+peqbgFGAn0AVHWZqo5T1b2eCc5eAeqKSJmAHIgxxpg8EdUCD/Wfvx2JNAOWqGpxr7LBQDtV7Zyr7gGgk6p+53ndApivqiV9bPda4E1VrXiS/fbHaRERFxfXfMqUKWd0HGlpaURFRZ3RNgpLUYoFilY8FotvRSkWKFrxWCy+tW/ffqWqtijQm1U1IAvOzJfbc5X1A1J81M0G6nm9jgcUT0L0Kq8CbAFuyksMzZs31zM1f/78M95GYSlKsagWrXgsFt+KUiyqRSsei8U3YIUW8Hs/kNdg0vj7DZnROLNIn65uNJDmOVgARKQcMBt4Q1U/LORYjTHGnKFAJph1QKiIxHuVNQFW+6i72rPOZz0RKYWTXGaq6v/5IVZjjDFnKGAJRlUPAdOBoSJSQkQuBroC7/uo/h7wgIhUFpFKwIPARAARiQZm4VzPeSQgwRtjjMm3QHdTvhuIBHYCHwJ3qepqEWkrImle9cYAnwI/AT8Dn3vKAK4DWgK3iUia11ItYEdhjDHmtPI94diZUNW9OPe35C5fhDM75rHXijM98xAfdd8F3vVjmMYYYwqBDRVjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLyzBGGOM8QtLMMYYY/zCEowxxhi/sARjjDHGLwKaYESktIjMEJFDIrJZRHqepJ6IyHAR2eNZhouIeK0fKyJrRcQtIn0CdgDGGGPyLNAtmNFABhAH9ALeFJGGPur1B64FmgAJQGfgTq/1PwB3A9/7NVpjjDEFFrAEIyIlgO7Ak6qapqqLgZnALT6q9wZGqupfqroFGAn0ObZSVUer6lzgqP8jN8YYUxCiqoHZkUgzYImqFvcqGwy0U9XOueoeADqp6nee1y2A+apaMle9xcA7qjrxFPvtj9MiIi4urvmUKVPO6DjS0tKIioo6o20UlqIUCxSteCwW34pSLFC04rFYfGvfvv1KVW1RoDerakAWoC2wPVdZPyDFR91soJ7X63hA8SREr/LFQJ+8xtC8eXM9U/Pnzz/jbRSWohSLatGKx2LxrSjFolq04rFYfANWaAG/9wN5DSYNiM5VFg2k5qFuNJDmOVhjjDFngUAmmHVAqIjEe5U1AVb7qLvas+509YwxxhRRAUswqnoImA4MFZESInIx0BV430f194AHRKSyiFQCHgQmHlspIuEiUgwQIExEiomI3dNjjDFFSKC/lO8GIoGdwIfAXaq6WkTaikiaV70xwKfAT8DPwOeesmNmA0eAi4CxnueX+j98Y4wxeRUayJ2p6l6c+1tyly8CorxeKzDEs/jaTpKfQjTGGFNI7LSSMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8swRhjjPELSzDGGGP8whKMMcYYv7AEY4wxxi8CmmBEpLSIzBCRQyKyWUR6nqSeiMhwEdnjWYaLiHitbyoiK0XksOexaeCOwhhjTF4EugUzGsgA4oBewJsi0tBHvf7AtUATIAHoDNwJICLhwCfAJKAU8C7wiafcGGNMERGwBCMiJYDuwJOqmqaqi4GZwC0+qvcGRqrqX6q6BRgJ9PGsSwJCgVdVNV1VXwMEuMzPh2CMMSYfQgO4rzpAlqqu8yr7AWjno25Dzzrveg291v2oquq1/kdP+Ve5NyQi/XFaRABpIrK2YOHnKAvsPsNtFJaiFAsUrXgsFt+KUixQtOKxWHyrW9A3BjLBRAEHc5UdAEqepO6BXPWiPNdhcq871XZQ1bHA2IIE7IuIrFDVFoW1vTNRlGKBohWPxeJbUYoFilY8FotvIrKioO8N5DWYNCA6V1k0kJqHutFAmqfVkp/tGGOMCZJAJph1QKiIxHuVNQFW+6i72rPOV73VQIJ3rzKcjgC+tmOMMSZIApZgVPUQMB0YKiIlRORioCvwvo/q7wEPiEhlEakEPAhM9KxLAbKB+0QkQkQGesrn+TN+L4V2uq0QFKVYoGjFY7H4VpRigaIVj8XiW4FjkROvlfuXiJQGxgMdgT3AI6r6gYi0Bb5U1ShPPQGGA3d43voO8PCxC/si0sxT1gBYA/RV1f8F7ECMMcacVkATjDHGmPOHDRVjjDHGLyzBGGOM8QtLMHng6UwwzjN+WqqIrBKRK4tAXPEiclREJhWBWG4UkTWeceY2eK6rBSOO6iLyhYjsE5HtIvJvEQnI/V4iMlBEVohIuohMzLUuWUR+9YyfN19ELghGLCLSRkS+FpG9IrJLRD4WkYrBiCVXnadEREWkgz9jOV08IlJcRN4Qkd0ickBEFgYxlhs8f1OpIvKLiFzr51hO+T1XkM+wJZi8CQX+xBl1IAZ4AvhIRKoHMSZwxnZbHuQYEJGOOJ0ybsO54fVS4PcghfMGsBOoCDTF+Z3dHaB9bwWex+nIkkNEyuL0oHwSKA2sAKYGIxac8fvGAtWBC3DuH5sQpFgAEJFawPXANj/HkZd4xuL8jup7Hv8ZjFhEpDLOeIsP4Nzn9xDwgYiU92MsJ/2eK+hnOJB38p+1PF2sn/Eq+kxENgLNgU3BiElEbgT2A98AtYMRg5dngaGq+q3n9ZYgxlID+LeqHgW2i8hXHB9myK9UdTqAiLQAqnit6gasVtWPPeufAXaLSD1V/TWQsajql971ROTfwAJ/xHC6WLyMBh7G+efA704Wj4jUA7oAVVT12KgjK4MRi+f5fq/f1+cicgiohfMPlD9iOdX3XBkK8Bm2FkwBiEgczthqQbm5U0SigaE4/90ElYiEAC2AciKyXkT+8pyWigxSSK8CN3pOdVQGrsTHGHUBdsLYep4/5A0EKPGdxqUE8SZlEbkeSFfVL4IVg5dWwGbgWc8psp9EpHuQYlkBrBGRLiIS4jk9lo4z7mJA5PqeK9Bn2BJMPolIGDAZeNdf/33mwXPAOFX9K0j79xYHhAE9gLY4p6Wa4TSvg2Ehzof+IPAXzh/qf4MUyzH5Gj8vUEQkAXgK5/RLMPZfEngBuD8Y+/ehCtAI53dTCRgIvCsi9QMdiKpm49xw/gFOYvkAuNPzxe53Pr7nCvQZtgSTDyLiwhl5IAPnwxeMGJoCHYBXgrF/H454Hl9X1W2quhv4F3BVoAPx/H6+wjlXXAJnRNpSONeHgqnIjZ8nIrWBL4H7VXVRkMJ4BnhfVTcFaf+5HQEygedVNUNVFwDzgU6BDsTT2eElnOlJwnGui7wjAZhc8STfcwX6DFuCySMREWAczn/s3VU1M0ihJOFcoP1DRLYDg4HuIvJ9MIJR1X04LQXvO3aDdfduaaAazjWYdFXdg3MBO+DJLpcTxtYTZ26kWgTvFOsFwBzgOVX1NVRToCTjDPm03fNZropzUfnhIMXj6/RTsD7LTYGFqrpCVd2quhz4DuefS785xfdcgT7DlmDy7k2cniWdVfXI6Sr70VicX2xTz/IW8DlweRBjmgDcKyLlRaQUTs+bzwIdhKf1tBG4S0RCRSQWZ/K6gJy39uyzGBAChIhIMXG6SM8AGolId8/6p3DmNPLbKdaTxeK5LjUPJwm/5a/95yUWnATTiOOf5a04M9eODlI8C4E/gEc9dS4G2gOzghDLcqDtsRaLOMNjtcX/n+WTfc8V7DOsqracZsHpzqnAUZym4rGlVxGI7RlgUpBjCMPpAbQf2A68BhQLUixNcQZE3YczYdNHQFwAfxeaa3nGs64D8CvOaZgUoHowYgGe9jz3/hynBevnkqveJqBDkH9PDYGlwCHgF+C6IMYyEFiPcxrqd+BBP8dyyu+5gnyGbSwyY4wxfmGnyIwxxviFJRhjjDF+YQnGGGOMX1iCMcYY4xeWYIwxxviFJRhjjDF+YQnGmLOEOPOl9Ah2HMbklSUYY/JARCZ6vuBzL9+e/t3GnJ9sPhhj8m4OcEuusoxgBGLM2cBaMMbkXbqqbs+17IWc01cDReRzz5Sym0XkZu83i0hjEZkjIkfEmbJ4oojE5KrT2zMPSbqI7BCRd3PFUFqcaY4PicjvPvbxlGff6Z5BJN/zy0/CmDywBGNM4XkWmIkzHtpY4D3PTIXHRp+dhTO2UyvgOuAivKbKFZE7gTE4g4cm4IwC/XOufTwFfIIzsu1UYLyIVPO8vzvO6Np3A/HANcAyPxynMXliY5EZkwciMhG4GWcgQG+jVfVhEVHgHVXt5/WeOcB2Vb1ZRPoBI3Cm4031rE/CmW8kXlXXi8hfOAOXPnKSGBR4UVUf9bwOxZlYrb+qThKRB3BGI26kwZtOwpgcdg3GmLxbCPTPVbbf6/nSXOuWAld7ntfHGd7ce4KmbwA30EBEDgKVgbmniSFnuHZVzRKRXUB5T9HHOLNDbhSRWTiTr81U1fTTbNMYv7BTZMbk3WFVXZ9r2V0I283PaYTcLRPF83esqn8CdXFaMQeBkcBKz+k5YwLOEowxhaeNj9drPM/XAI0989AfcxHO3+AaVd0JbMGZhKvAVPWoqn6uqv8EWuLMb3LxmWzTmIKyU2TG5F2EiFTIVZatqrs8z7uJyHKcyZh64CSL1p51k3E6AbwnIk8BpXAu6E9X1fWeOv8HvCIiO3BmKS0OJKvqyLwEJyJ9cP6mv8PpTPAPnBbPb/k8TmMKhSUYY/KuA7AtV9kWoIrn+TNAd5wZPXcBt6kzlzqqelhELgdexenZdRSnN9j9xzakqm+KSAbwIDAc2At8kY/49gMP43QmCMOZkbGbqm7MxzaMKTTWi8yYQuDp4XW9qv4n2LEYU1TYNRhjjDF+YQnGGGOMX9gpMmOMMX5hLRhjjDF+YQnGGGOMX1iCMcYY4xeWYIwxxviFJRhjjDF+8f+KPI6pmgiOiAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "id": "yJIAA4wP5jlC" }, "outputs": [], "source": [ "np.random.seed(43)\n", "\n", "series = generate_time_series(1, 50 + 10)\n", "X_new, Y_new = series[:, :50, :], series[:, 50:, :]\n", "Y_pred = model.predict(X_new)[:, -1][..., np.newaxis]" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "scrolled": true, "id": "9Q83vZbA5jlC", "outputId": "fca5256c-633c-415a-8d97-38d8a15b4f6d", "colab": { "base_uri": "https://localhost:8080/", "height": 293 } }, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEUCAYAAABpmDIHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUVf748fdn0hMIpNFJoYReQxMEE9eGoiBgVo2oK4prXde+6/q1/Fxd0fWxIYqwVgRRQRDWskhCDyX0XtLokEJJQkiZ8/vjTmAICaRMuRPO63nmSXJuO5cM88k95XNEKYWmaZqmuYvF3RXQNE3TLm86EGmapmlupQORpmma5lY6EGmapmlupQORpmma5lY6EGmapmlupQORpmma5lamDEQi8qiIrBWRMyLy+SX2/auIHBaRkyLyHxHxs9sWLSLJIlIkIjtE5BqnV17TNE2rFVMGIuAg8Brwn4vtJCLXA88DfwCigHbAK3a7zADWA2HAC8D3IhLhjAprmqZpdSNmzqwgIq8BbZRS91az/RsgUyn1d9vPfwCmK6VaiEgssBkIV0qdsm1fatv+sUtuQNM0Tbskb3dXoJ66AXPtft4INBeRMNu29IogZLe9W1UnEpEJwAQAf3//uMjISOfU2ASsVisWi1kfhutP35/nasj3Bg3//nbt2pWjlKp1q5OnB6JGwAm7nyu+b1zFtortras6kVJqCjAFoFOnTmrnzp2OramJpKSkEB8f7+5qOI2+P8/VkO8NGv79iUhWXY7z9NBcAATb/Vzx/akqtlVsP4WmaZpmGp4eiLYCvex+7gUcUUrl2ra1E5HGlbZvdWH9NE3TtEswZSASEW8R8Qe8AC8R8ReRqpoRvwTGi0hXEWkK/AP4HEAptQvYALxkO/5WoCfwg0tuQtM0TasRUwYijIByGmNo9l227/8hIpEiUiAikQBKqV+AiUAykA1kAS/Zned2oB+QD/wLGKuUOuayu9A0TdMuyZSDFZRSLwMvV7O5UaV93wHeqeY8mUC842qmaZqmOZpZn4g0TdO0y4QORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZUORJqmaZpb6UCkaZqmuZVpA5GIhIrIHBEpFJEsEbmzmv1+ti0fXvEqEZHNdtszReS03fbfXHcXmqZp2qWYcqlwm0lACdAc6A0sEJGNSqmt9jsppYbb/ywiKcCiSue6WSm10Il11TRN0+rIlE9EIhIEjAFeVEoVKKWWAfOAcZc4LhoYCnzp7DpqmqZpjmHKQATEAmVKqV12ZRuBbpc47m5gqVIqs1L5dBE5JiK/iUgvB9ZTc6Dle3J4+9edpGXlu7sqmqa5kCil3F2HC4jIUOA7pVQLu7IHgCSlVPxFjtsDvKaU+tyubAiwDhDgL7ZXZ6XU8UrHTgAmAERERMTNmjXLYfdjNgUFBTRq1Mjd1ThPcnYpX2wrAcDHAs/196dDiFedzmXG+3Okhnx/DfneoOHfX0JCQppSql9tjzNrH1EBEFypLBg4Vd0BInIl0AL43r5cKbXc7sc3ROQejOa7nyrtNwWYAtCpUycVHx9f17qbXkpKCma5P6UUn6/I5Kvt286WlVnhTNMo4uM71OmcZro/Z2jI99eQ7w0a/v3VlVmb5nYB3iLS0a6sF7C1mv0B7gFmK6UKLnFuhfF0pLlZ4ZkyHpuxnld+2kbfyBD8vM+9HftHh7ixZpqmuZIpn4iUUoUiMht4VUTuxxg1NxIYXNX+IhIAJAK3ViqPBNoCazCC7mNAOLC88jk015q34QAv/7SN/MISnruhMw8Oa8f6fcf5amUmP244yKETxe6uoqZpLmLWJyKAh4EA4CgwA3hIKbVVRIaKSOWnnlHAcSC5UnljYDKQDxwAbgCGK6VynVpz7aJ+XH+Ax2duIK+wBB8vCwNiQrFYhLioEN5J7E1s80ZMSt6D1Wq+/ktN0xzPtIFIKZWnlBqllApSSkUqpb6xlS9VSjWqtO8MpVSUqjTyQim1VSnV03aOMKXUH5RSa115H9qF3vt999nvy61WUtPP/V1gsQgPx3dg15EC/rf9iDuqp2mai5k2EGkN04o9OWTkFOJtEbwEfLwtDGoXdt4+I3q2JDI0kEnJezDjqE5N0xxLByLNZaxWxRs/76B10wC+Gj+AJ6/rxPT7BxEXdf7ABG8vCw/Ht2fT/hMs2Z3jptpqmvNNnw7R0WCxGF+nT6+6rKEz5WAFrWGav/kQmw+c4J3EXlzRPpwr2odXu+/ovm147/fdTFq0h6tiI1xYS01zvOnT4YUXIDv7KiIj4Z//NMonTICiIuP7rCz4059ABEpKzpVNmGB8n5Tk+nq7in4i0lziTFk5b/26gy4tgxnVu/Ul9/f1tjBhWDtWZ+axKl2PLdE81/TpRjDJygKlhKwsuO8+uP/+c0GoQmnpuSBUoajICGINmQ5EmktMT81mX95pnh/eGYulZtO4bu8fSViQL6//dzuTkvfo1D+aR3rhhQsDTkkJFNdihkJ2tmPrZDa6aU5zupPFpXywaDdXdghnWMfqm+MqC/D1Ynj3lny9KovNB07g622psk9J08zMEUEkMrL+5zAz/USkOd3HKXvJLyrl+eGdEaldUouwRj4AWBWUlJ0/1FvTPEF1QSQsDAIDzy/z8QFf3/PLvLzO9Sk1VDoQaU71v61HmLIknaEdw+neukmtjx8W2+xs6h+loGeb2p9D09zpn/88P+A8w0Ru8EvmvfdgyhSIijIGKERFwWefwX/+A7c3T+ZZJtK4MZSXQ58+7qu/K+hApDlNWlY+f56eRplVsTojr059PHFRIXzzwCBu69cGi8Ani9MpLbc6obaa5hxJSfYBR7GveX/m+CaS1CqZpCTIzASr1fialARJrZKZUZ7Im4v6k54OjRvDiy+6+y6cSwcizWlW7s2h3Jamp6y87s1qcVEhvDW2F2+M6cmyPTn8bfZmPdFVM7+JEyHZyDpWEXAWLVrMjMMJ+M+dBYmJ8M47xn4VkpON8lmzICGB8HB46imYPRvWrHHPbbiCDkSa0zQP9gfAUk0GhdpK7NeWv/yhI9+n7ee5HzbpkXSaufXvbwSV5MopMIGEBPjb3+Dpp8HbNmasUhCq8OSTEB4Of/+7i+rtBjoQaU5zqrgMgAnD2jlstNsT13QkPjaCWWv38/avO0mamqqDkWZOCQlGUKkqGCUnwxtvwOOPG+O7//jHKoMQGE1zf/sbLFwIzZs3zIwLOhBpTrMmM482IQE8P7yLw4Zciwh9bedSQKkeSaeZWVXB6Jdf4JZbIDQU3nvPmFA0axa0b39BEKoQGgoJJHPP0YkodS7jQkMJRjoQaU6hlDFAYUBMqMPPPaRDOL5exjBwL0v9m/w0zakqgtHYsfR55BG48UYoKDDGZT/+uBFlWrWCVauMbVX0f/76fDLfksga+p8ta0gZF3Qg0pxi77FCcgtLGOiEQBQXFcK0e/sjwMg+rfQEV838EhIgNpYm27ZB587G09GHH8I338D33xsjGfr0gZ9/hvh4YxhdheRk3j+SSCKzSOH8J6aGknFBByLNKVZn5AEwIMY5TytDO0bQrXUw+/KKLr2zprlbcjLs2sW+MWPg2DFYt87oF6roE/LxgbVr4corYckSGDDASDxnG8DwePMLgxA0nIwLOsWP5hSrM3KJaOxHdFjgpXeuo4ExYXydmsWZsnL8vL2cdh1Nq5eK0XDff89eEdoOHmyMlnv77fP7hCwWIwjdeKPRjxQTA2fOwKxZjDiYwLwJ5+esCwxsOBkX9BOR5nBKKVbZ+odqm9KnNgbEhHKmzMqm/Secdg1Nq4u1mXlMSt7DzhnzLhwNV1ZmBKE33rhwNJ2I0TzXvz8cOAC9e0NCwtlJsRVPQAEBxs8NZWkI0wYiEQkVkTkiUigiWSJyZzX7vSwipSJSYPdqZ7e9t4ikiUiR7Wtv193F5Wl//mkOnShmQLTj+4fsVZy/ohlQ08xg2Z4cbvt4JcumfEfE/ePY+f608598nn3WmBx0saHdGRkQG2uM2Z4xAzCCTlYWPPywEa9Gj3bhTTmZaQMRMAkoAZoDScBkEelWzb7fKqUa2b3SAUTEF5gLfA2EAF8Ac23lmpOsyazoH3JuIAoJ8qVT88as0oFIM5GpS9JRQM/Du3hs5PMsbNG16h0rRtPZp0ywn9S6cCH4+8O998KiRWd3GTnSaKJbuNCpt+FSpgxEIhIEjAFeVEoVKKWWAfOAcbU8VTxGP9i7SqkzSqn3AQGudmR9tfOtzsgj2N+bTs0bO/1aA2JCScvMo0znn9NMoLi0nA37j2MRmDpoLGnte198ekFCgvGEBBdmVmjb1mi+KymBUaPOPjnFx0NwMMyd6/z7cRWzDlaIBcqUUrvsyjYCV1Wz/80ikgccAj5USk22lXcDNqnzE5NtspX/Yn8CEZkATACIiIggJSWl3jdhVgUFBU69v5StRbRrbGHJksVOu0aFxqfLKCwp58v5ybRrYgxYcPb9uVtDvj9Pv7fk7FKOF5WS1MWXM2WKzqFenMrYSEqGsb26+2u6fj1dX3mFbS+9xHERsO0TuW0bLVq3xvvUKWTUKLa+/DLH+/QhLq4rs2c35c47VxC6cT2Nd+xg3x13uO5GHU0pZboXMBQ4XKnsASClin27Aq0AL2AwRjC6w7btRWBmpf2nAy9f7PqxsbGqIUtOTnbauY+cPK2inpuvPlm8x2nXOO96Jy68njPvzwwa8v158r2VlpWroW8uUiM/XKasVmuV+1R7f2++qdSiRReWL1qkVNOmSokoNXKksZ9Savp0pUCpLR8sUio8vOpj3QBYq+rwmW/KpjmgAAiuVBYMnKq8o1Jqm1LqoFKqXCm1AngPGFvb82iOsSbDyPvmrPlDlTUL9icmPEgPWNDcbsHmQ2TnFfFwfPvajxZ99tmq0/skJBipt/38jLa4+HjAGOH9B0syUc9WnZ/O05g1EO0CvEWko11ZL2BrDY5VGP1A2PbvKee/K3rW8DxaHazOyCXAx4turSrHf+cZGBPK6ow8rFa9NITmHlar4qPkvcQ2b8Q1XZo79uQJCfDdd8Y8ozvvhLIymq5P5nuvRB4J9/wgBCYNREqpQmA28KqIBInIEGAk8FXlfUVkpIiEiGEA8DjGSDmAFKAceFxE/ETkUVv5osrn0RxjVUYecVEh+Hi57q01ICaUk8Vl7DisH3Q191i04yg7j5ziofj2WCxOmDs3YgS89BLs3QvDh0NiIosenMWX+xLYtevSh5udKQORzcNAAHAUmAE8pJTaKiJDRaTAbr/bgT0YzW1fAm8qpb4AUEqVAKOAu4HjwH3AKFu55mAnikrZeeSU04dtV1ZxvdUZOgu35npKKSal7KFNSAA392zlvAu9+KKRoXvhQhg/nrinjSehefOcd0lXMW0gUkrlKaVGKaWClFKRSqlvbOVLlVKN7Pa7QykVpoz5Q52VMUTb/jzrlVJxSqkApVRfpdR6V9/L5WJtVh5KOX/+UGVtQgJp3TSA1Zm6n0hzvdT0PNZnH+fBq9rj7cyWgJQUyLX9sfXhh0SlJ9OrV8MYxm3aQKR5ntUZefh6WejdtqnLr13RT6T0EuKai32UsofwRn7cFtfGeRepmGM0ezYMG2YkmktM5C89k1mxwsij6sl0INIcZlVGHr3aNsHfx/UJSAfEhJJTUMLeY4Uuv7Z2+fpuzT6W7s5hePcWznvfV57o+vzzRuQZP567fkpkmDWZBQucc2lX0YFIc4jle3LYtP84bUOcl237YgbaZq/rYdyaq6Rl5fPc7E0AfLd2n3OWrK8chABuuAF69YJ58/D+bibfSSIzH0z26CXEdSDS6i0tK58/fbYGq4L5mw465z/kJUSHBRLR2I9VesCC5iK/bz9CxYyB0nInLVm/Zs2F84REjKei7dtZsuAUt8ssepas8eglxHUg0uotNT2XEluut3Krcs5/yEsQEQbEhLIqXfcTaa5xqrgMAIuAj7eTlqyvbqLr2LHQrh1NPnqD363xvMWzZzd54hLiOhBp9dbdNnlVcOJ/yBoYFBPK4ZPFzNpZ4panMu3yYbUqlu4+RpeWjXnquk5Mv3+Qa5esf+cduOUWepesJp6UCzZnZ2M0602c6Lo61YMORFq97TpiTOu6+4po1/+HtNPIzweAXzLLSJqaqoOR5jSp6blk5hbx4LD2PJLQwfXv+f794euvOS4hPM+/Ltj8x2a2vqX+/V1brzrSgUirF6tV8VVqFv2jQ3hlZDe3BSGAA8eNdZQVUFrmpDZ7TQOmr86mSYAPN3Rv4Z4K2NYyCvIt4Xp+ow/rzm66wS+Zz4o8KwedDkRavSzedYzsvCLuviLa3VXhivbheNnSq/h4ua+JUGvYcgrO8NvWw4zp28YtUxXOSkjA57sZKIQPfJ4CYGRwMnN8E/Gf6zlBCHQg0urpi5WZNGvsx/Xd3PSXoZ24qBD+b4SxGubD7mgu0S4LP6Ttp7RccceAtu6uCtx8M3LH7QwuTWGq/6N8ecbzghDoQKTVQ2ZOISk7j3HnwEh8vc3xVrprUBRN/YStB0+4uypaA6SUYsbqbPpHh9DRBSsQ18g77yAijC+exDSfh1DxnhWEQAcirR6+Ts3C2yLcOSDS3VU5y8si9G/hRfLOY5wsLnV3dbQGZqVtkMKdA83znmf7dvDxoczHn7sKJpP1ebK7a1RrOhBpdVJUUsastfu4oXsLmgX7u7s65xnY0puSMiv/23rE3VXRGpgZq/fRJMCH4d1bursqhorMC6+/jndpMZ9yP80eSzTKPYgORFqdzN1wkJPFZdwzONrdVblA+yYWWjcN4KdNB91dFa0ByS04w69bDjO6b2v3DlKoYJ/+58knITaW6/2X8FqPWUa5BwUjHYi0WlNK8cWKTLq0DKafCQcEiAg392rFst055BXqpac0x5i97gAl5VbuMENTdOUcdCLw4IPEFa9g0cYwyr7xrGCkA5FWa9NXZbPj8Cmuig3n/FXYzePmXi0psyp+2XLY3VXRGoC0zDw+StlD5xaNiTXDIIWqctDdcw/lPn6MO/0JqQHGPCPWrHFfHWtBByKtVtKy8vm/uVsA+Hx5pmmzF3RtGUy7iCB+2qib57T6ScvK5/ZPU8kvKmXvsQJzvOerykEXFkbZ6ETG8RWLFxQY2599turjTca0gUhEQkVkjogUikiWiNxZzX7PiMgWETklIhki8kyl7ZkiclpECmyv31xzBw3Twm0uyDjsACLCzT1bkZqRy9GTxe6ujubBZqzOorTceNNb3ZTUt6b8Hv8zwZzC67sZ7q5KrZg2EAGTgBKgOZAETBaRblXsJ8DdQAhwA/CoiNxeaZ+bbUuJN1JKXefMSjd0h20f6k7NOOwgN/dqiVKwYPMhd1dF81Drs/OZt+EQAnh5wHueK67gcLMeXLP3E0540FQ6UwYiEQkCxgAvKqUKlFLLgHnAuMr7KqUmKqXWKaXKlFI7gbnAENfW+PJQVFLGoh1HGRgT6p6Mw7XUoVljurQM1s1zWp3sPVbAfZ+voXkTPz69px9Pmv09P3EipKRw8o4/0480Nkxde+E+Js3I7e3uClQjFihTSu2yK9sIXHWxg8ToOR8KfFJp03QRsQDrgWeUUhurOHYCMAEgIiKClJSUutfe5AoKCup0f4uySzlxupQ/NCsiVvZzKmM/KRmOr1992d9ft8YlfL+rlO/+u4iIQFP+3VVrdf39eQKz3Ft+sZXXUospsyoe7eaN95HtdBPq/Z535v019fGh6623UvDkcxQQROkH/yYl7sFz29evp+srr7DtpZc4boJ/4/MopUz3wggmhyuVPQCkXOK4VzAClp9d2RAgAAgE/gYcBppe7DyxsbGqIUtOTq71MeXlVpXwdrIa8f5SZbVaHV8pB7K/v+zcQhX13Hw1OWWP+yrkYHX5/XmK2tzbb1sOqw8X7VJrM/McWofFO4+qfv/vN9Xphf+qTfuOO/TcTv/dLVqkVHi4Sg2/SRVKoFL5+eeVq0WLnHp5YK2qw2e+WZ+ICoDgSmXBwKnqDhCRRzH6ioYqpc5UlCulltvt9oaI3IMR6H5yXHUbvsW7jpF+rJB3/9jbtEO2q9I2NJCOzRrxn2UZ9I8ONW+zilYrU5bs5fX/7kAAP5899W4yU0qx60gBny3P4Ns1+1CAr5fl7MrDHsO2PET360cTqIp4LORrcpp347MicydDNWsg2gV4i0hHpdRuW1kvYGtVO4vIfcDzwDCl1P5LnFthDHDQamHasgyaB/txYw+TpDapobSsfDJyCimzKu78NJVvHjBxG79WI0UlZXy4aA9g/Gcusa09Vdvfa1pmHrPXH6DgTBkb9h0nK7fovO3l1rqd192mH0zgS2Yzn+t4kVeQI3Cr3yzuOphAkrsrVw1TNporpQqB2cCrIhIkIkOAkcBXlfcVkSTgdeBapVR6pW2RIjJERHxFxN82tDscWF75PFr1dhw+ybI9Odx9RbRpsmzXVGp6LlajifbsB5bm2d5duJuTxWX4eJ37e3JQTGitzpGWlU/ilFSmr8pm7oaDNA3w4bVR3Zl6dz/8fSyeMUKuGi+8AL+VJrCAG2lGDr9wPb+cSeCFF9xds+qZ+VPlYYy+naPADOAhpdRWERkqIgV2+70GhAFr7OYKfWzb1hiYDOQDBzCGdw9XSulPo1r4bFkm/j4WU2XZrqlB7cLOBk+LiEd+sGjnbDt4kmnLMrhjQFtmTriCqzs3w6pg//HTtTrPlCV7KbdNiPMSuK5bC+4aFMU1XZsz/f5B5h8hdxHZ2RBPMkNYQQGBJPId8SSTne3umlXPrE1zKKXygFFVlC8FGtn9HHORc2wFejqlgpeJnIIzzNlwgLFxbQgJ8nV3dWotLiqE6fcP4m+zN5FXWOKRHyyaodyq+PuczTQN8OG5GzrTNNCXT+/ux+iPlvP/5m/jqtgImgZe+j26Ym8OC7cdwSJGG33lJ5+4qBCPfp/8sVky7x9JJJFZxJPCS7zKD4zhkWY/AObsIzLzE5FmAtNTsykps3LfkGrjvenFRYVwx4BIcgpK2J9fdOkDNFP6ZlUWG/Yd58URXc8GHC+L8ProHuQXlfLmLzsueY6MnEIe+nodMRGN+OK+AR795FOl5GQ+K0rkbr9ZpJDAZB6iBB+WWOL5rMi8SVB1INKqlZqewydL9tInsikdmjW69AEmNjDG+It3dUaem2ui1cXRk8VM/GUnV3YIZ2TvVudt69aqCfcNiWbG6n0X/f2eKCpl/OdrsAhMu6cfQztG8EhDWlLelpHbf+4s7pqWQGQkHKEF33vfzk2+/8N/+n9Mm5FbByKtSmlZ+YybtpqiknK2HjhpjkSP9dCpRWOC/b11IPJAaVn5jPvPak6XlfPaqO5VTh/467WxtG4awF9nbeD933dd8H5dnZHLLR8uIyuvkI/viiMqLMhV1Xcdu4zcSUmQlQVjxsCMiL/gU1wAe/aYNiO3afuINPdasTfnbKJHTx3Gas/LIgyICWWVDkQeJS0rnzumrKSkXOFtEXILS4gOvzCIBPp6c+/gaP753+2887/dfLBoD38aEkOrJv5k5hby5cosrAp8vARvrwb693cVmbaHDYO//BBHcf8r8f/gA9i925RziRrob0SrrzLbRD5PSG5aUwNjwsjIKdTZuD2EUorJi/dSYvuDSKmLZ762n3xaWq6YsiSdl3/axucrss5mjDd79mxHGzbM+Lr6ir9ARgbMn+/eClXD4YFIRN4XkQvuVkSCReRlEeliV/aEiGy25YHTTKKs3MpPGw8RFRrIk9fGNpjO3AG2uSb6qcj8jp4q5r7P15wd3VaTeT2D2oXh72PBIuDnbWHq3f1I+8c1fDV+AP7enj03qK569IAX/SeStqcpREbCu+9WvaObk6E6tGlORNoDfwYGV7G5H/ASxkTVCp9gZES4B/jMkXXR6m7exoOk5xTy8V19uaG7Z2VSuJhurYIJ8vViVUYuN/dqdekDNJdKy8pn/t4StrGHqUszKDxTxiu3dKN7q2BSM/IY1C7son8QVQzVT03PPW/foR0jmP7AheWXAy8vKO3Vn3t+TYTxo2HKFNi0CXrazWqxX3bcTRzdR/QEsFEpVUX+cfoAZ4BtFQVKqdMi8iXwNDoQmUJZuZUPFu2hS8tgruvawt3VcShvLwtx0aF6wIIJpWXlk/RpKsVlVti9k3bhgcx6cBAdmhnLcsdF1yxzQnVzgDx9blB9hIxOYMyqWfz+3W1Y/Pzgvfdg2jRjo30QcmPfUY2axESkg4iUisirlcon21ZG7ScifsBdwDdVHL8deBvwA0pFRInID7bNM4GuIlLVU5RmJy0rn0nJe5w6gm3uhoNk5BTylz90xGJpeCn5BsaEsutIAXmFJe6uimYnNT2XM2VGH48Ao/q0ORuEtPoZNgxSSGDZX74DEfjqKzh2zDRBCGoYiJRSe4CpwBMiEgYgIv8H3AfcansCGgQ0BZZWcYq7gXSMjNdX2F5P2rZtwMiqfUPdb6Phq/iL8d+/7SRpamqdg1FF80dVxxtPQ7vp2jKY67s1r2+VTWmgrZ9IPxWZywC7XHF+PhaGdAh3Y20alr59ITAQfshLgMmTobTUCEAmCUJQu8EKrwJewPMicj9Gf884pdRC2/ZBGMlwN1Vx7EagDbBIKZVqe2UBKKWstu2D6ngPl4XU9ByKy6xYFZwprVvyTmMobCo/7C6tMpj9uOEgmblFPHFNR49a6qE2erZpip+3hVUZl8/IKU9woqgUBQxq6dVgBseYha8vXHEFLFkC3HsvREdDSgo8+KApghDUIhAppQ4B7wKPAR8Djyul7Hu3WgEnlVJVtXl0A3yBddWc/pjteK0aJWXq7PcKKDxTWutz/Lb1MCXlVhRQXGrlq5WZFYsHnn0a6tYqmGu7NsynIQBfbwt9I0P0E5HJfLEykxbB/tzfw08HIScYNgw2boSCn5Ihz/be/+AD02RZqO2w6d0Y/TwrlVKTKm3zxxiMUJW+GJ+fG6rZfhoj07ZWhZyCM3y5MpPY5o346zUd6d22KRj1o2IAACAASURBVB8vTmfBpkM1PodSirW2JyCxvX7ccJAxk1ewJjOPd3/fTVZuETf3atVgn4YqDGwXyrZDJzlxuvbBXHO8PUcLWLo7h7sGReLdAPslzWDYMLhKJeNzVyJ8/z20bAmdO5sm5U+NA5GI/AFjuPVKYIiIVM5qnYvRR1SVPsBepdTJaraHAjk1rcvl5qV5Wyk8U86kO/vyl2timWFb3O2Jb9eTvPNojc4xf9Mh0rLyuXdwFGM6+vDtg4N4c0wPDhw/zW0frzy70Ni7Cy9Mj9LQDIgJRSlIy9JPRWbw1cpMfL0s3O6By4x4iiuKk5lFIp8PnwXXXgsPPGCk+nnvPVMEo5qOmusLzMEYsBAPZANvVNptB+ArIm2qOEVX7IZtVyEG2FmTulxuft16mAWbDvHY1R3o2NwYRRTg68W0e/sT27wxD32dxtcrsy46mu5EUSmv/LSVHq2b8I+bujKivS8DYsL4Y/9IUp5OYGjHcx3DpZfB4nF9I0Pw9bKwKl0HInc7VVzK92n7GdGzJeGN/NxdnYYpORm/cYm82m0Wn2fZ+oQeeAAsFmNO0axZbg9GlwxEItIB+Bn4DXjM1gf0CnCjiAyz23WJ7euAKk5zHOglIteLyKCKkXe28zcFYu2O12xOnC7lxR+30KVlMH+Ob3/etmB/H768bwBhQb78Y+6Wi46me/2/28kvKuWN0T0uyLMV4OvFE9fEevyqlLXh7+NFr7ZNdIYFE5i97gCFJeXcMzja3VVpuGzJUBvfksDatVBYCLRpA7fcYswnGjzY7clQLxqIRKQFRgDaDiTZRrgBfInxBPSvin2VUpnAauDmKk71f8AR4EeMpr0udttuAkownrg0O68v2E5uYQlvje2JTxWJGsMa+Z3NEFAxmm75nmPn7bNyby7frt3H/VfG0L11kyqvUzEjvcGtzXIRA2JC2XzgBIVnytxdlcuW1ar4YmUmvdo2pVfb6lr1tXp79llISGDYMCgrg9RUW/lDD0FOjtFnlJBQZdJUV7loIFJKHVZKtVNKxSulztiVlyuluiilKk9CnQyMFpHASufZopQaqJQKUEqJUmqZ3ea7gO8qL98tIqEiMkdECkUkS0TurKqOYnhTRHJtrzfFrrddRHqLSJqIFNm+9r74P4k5fLYsg2/X7uOWXq2qDSAA13ZtgZ9tKWwFfLt2P+uzjaei4tJyXpizmbahATxxTexFrxcXFdKw1ma5hIExYZRbFS//tLXB94mZ1fK9OaQfK+TewVHursplYfBgozVuSUXb0x/+AB06GHOL3MzRyUa/Bg4CD9dkZ1tQuBqjqa+ySRhPSs2BJGCyiHSrYr8JGEuK98JYFvxm4EHb+X2BubZ6hQBfAHNt5aa1aMcRXp1vdKn9vPnQRT8o46JC+OaBQTxzfSdeHNEFZVWMmbyCx2esY9y0VaTnFPL6rT0I8PVyVfU9QkXWiO/X7q/XBGGt7r5YkUl4I19u7NFw8hmaWXAw9O5tC0QTJ8LixfDnP8Py5bB584UHuDARqkMDkVKqDPgTUNP1mFsA99oyN5wlIkHAGOBFpVSB7QlqHjCuinPcA/xbKbVfKXUA+Ddwr21bPEY+vXeVUmeUUu9jjFy+ulY35kInTpfy/A+bqZg1VFp+6cEDFU8z469sx69/HcY1XZozb+Mh1mTm4yVCoK9edqqyjfuOA8ZT5JlSK79uPezeCl1msnOL+H3HUe4YEImft/4jyVWaNTPmsl79XH9yr0kkZX978PO78KmoIv1P//4uqZdUTGg0ExHpAyxXSgXalT0NXKWUurnSvieA65RSq2w/9wOSlVKNReSvtm3D7fafb9v+70rnmYDxdEVgYGBcUVFNY6njiI8fzRL/H34tY0FZQSwoazlHZr5AycEdNT5P8KDbaDr0LsTihbKWc3zp15xM/c6JNfc8vq060/z2fyLePoCAtZyCzf/jxIpZeDUOwz+yB8XZm2v1767VXNiIpwjqchXHZr/G6b2r3V2dy8QdwH8wpnxCPMnM4jbS8GEIh2kFFGD89T4LSARSan+RNKVUv9oeZNY/lRsBleccnQCqyoLYyLbNfr9Gtn6iytuqPY9SagowBaBTp05q507XjiY/U1bO/V+sZfmeHCbd2Zdmwf7n0tb/e3utzpWWlU/S1FRKy6z4+Pkwe+YnxEWdS4KRkpJCfHy8g+/APGp6f2lZ+aSm59KxWSOW7s7hW19fmvYdDghKKXy9LaYcvOHpv79V6bn8cYrRYx55x8vn/Rt7+r1dijvvLzraWD78bF1IIJHv+FFG01jBqcmToVOnsznokuuQ/qeuk+HNGogKgOBKZcEYyVEvtW8wUKCUUiJSm/O4TVm5lce+Wc/S3Tm8NbYnw21t5nX9AKxuXRbtfPZLA1zXrQUPxbfn4elpbNhn/O1SMadK//s51rdr9539Xv8bu0529oVlKSRwq/qBRd7Xw0svgdXqlkSoZl0ZdRfgLSId7cp6AVur2HerbVtV+20Fesr5YbpnNedxi7WZeYz8cDm/bTvCyzd35bZ+bR1y3sttFJwjtGoawIsjulGRZeZymFPlDkdPGgNwL5d5a2YRWU3iivSoq2H4cDh61BhJ54ZEqKYMREqpQoyVXF8VkSARGQKMBL6qYvcvgSdFpLWItAKeAj63bUsByoHHRcRPRB61lS9yZv1rKi0rn9unpLL10Em8LUKPNnouhbvFRYXw4k1dARh/ZYwO5A5WVFJGWlY+13ZtflnNWzODf/7TWA7CXmAgTE1KhpUrISTEmFP0++8ur5spA5HNwxiJUI8CM4CHlFJbRWSorcmtwicY6xxtBrYAC2xl2LJAjMJYD+k4xvpJo6rJEO5yqem5lFmNwSJKqQafWsdT3Dskms4tGrNw21HMOJjHk/1v2xFOl5Zz/5Ux+ondxZKSjJXCo2zTtiwWmP14MtdMsa1L9MEHUF4Ot97q8nQ/pg1ESqk8pdQopVSQUipSKfWNrXypUqqR3X5KKfWsUirU9npW2X16KKXWK6XibJNp+yql1rvjfqrSrZXRfSXoJgozEREeGNqOnUdOsWS3zsXrSD9tPEjLJv70r+HS35pjJSVBZibMmwfDrMnEf2S3ON7tt0O3btC0qctzz5k2EF0OikuNjEl3DIjUTRQmc3OvVjQP9mPq0nR3V6XBOF5UwuJdx7i5V6sGuQy9J7nBL5nvJJGJcXYDE7y84NVXYd8+uO8+lwYjHYjcKDU9F38fCy/f0k0HIZPx9bZwz+Bolu7OYfuh6lYvqZ20rPyLZklv6H7ecpjScsUtvfQamO7ms2ENX98yi9dXJnDS/u19660QF2c8JU2f7rJEqDoQuVFqei79okLx9da/BjNKGhBFgI8XU5dm1PtcaVn53PFp6kWzpDd0czccoF1E0Nkmac2Nnn2Wgc8lUFwMc+zTTb/1Ftx2m9F+t3dv1YlQnZD6R38CukleYQk7Dp9iUDvdVm5WTQJ9SOzXhnkbD3DkZHG9zvXj+gOUlFmxKqNJdsHmmq+u2xAcPlHMqow8brkMVgD2FIMGQbt28PXXdoX9+8Pbb0P37vDaa3D69PkHOSn1jw5EbrI6wxghpwcomNt9V8ZQZlV8sSKzzuc4duoMCzYdOrtEOxjZ1Z+YuZ55Gw9cFs118zcdRCl0s5yJiMBddxmjtQ8etBUmJBjNcvv2GYUffXTugIog5IQJrzoQuUlqeh4BPl701HOHTC0qLIjru7Zg+qpsikpqv3ZRSZmVR6avo6i0jLdu68nT13fis3v78eBV7fl5y2Een7GBt37dyZ2fNuzmurkbDtKjdRPaRTS69M6ayyQlgVIwY4ZdYUKC0V7n42MMXjh1yqlBCHQgcpvU9Fz6RYfo/iEP8MCwGE6cLuWxGetrHSxeW7CN1Zl5vDmmJ2Pj2vJIQgcSOjfn+eGduX9ozNknpDNlVhZuP+L4yptA+rECNh84wcje+mnIbGJjYcCASs1zYASbd9+FkyfhppucGoRAByK3ONc/pJvlPIMgAr9vP8ptH6/ghTmbWZ+dT1m59aIj4Wat3ceXK7O4/8oYRvZufcH2qzs3x8/Hcjal0Pdp+8jIKXT2zbjcvI0HEYERPXUgMqNOnWDDBmOCa3S0MVgOgIcfNuYVLV0KY8Y4NfWPWZOeNmjn+of0QAVPkJqei2CsXWRVMH1VNtNXZRPg48WZsnKUAh8vC5/9qT9DOoSTlpXPj+sPMGNNNoPbh/H88M5Vntc+OW1EI1/+9ctOxk5ewRf3DbjoqryeJC0zjy9WZtK1ZWNaNPF3d3W0SqZPN7L6gNFEl5UFEyYYPye1SobDh408QNOmwdixcM01TqmHDkRuUNE/1KO17h/yBIPaheHrbTGW1fC28FFSX06XWPl06d6zmbpLyq3cNXUVkWGB7M8/TblVIcADw9rh7VV9w4N9BvB+0aGMm7aa26ek8vR1sRSWlHt09vS0rHzumLqKkjIrBcVGjjlPvZeG6oUXLhwYV1QE859KJqk8Eb77DnJzjSHdI0fC/Pm6j6ih0P1DnqXiyaUiSefVnZtzU8+WvDiiG/4+FrwEfL2Esf3aIEC5LX+gCGw7WPPJsO0iGvHDQ4MJCfTh5Z+28favnj3nKDU9l9IyI3uI1apzKZpRVUtDxJPM+0cSWThhFtF/SuC529JZ6XcV5cUlRhNdVdkW6jm3SH8SupjuH/JMVS2rYR+gZky4grfG9uLfib3PBac65A9s0cSfUX2M/iSFMerOUz/AB7ULQ/SSGqZWeWkIY9XWRBKZxY1vJZCVBavpT8czWzhlDeSEX8SFqX8cMLdIByIXW5Wu+4caksoBqvLTU12aouI7NcPP9rSsFPTz0Oasnm2a4G2x0Deyqc6laFKVl4bozxru8pnFCt8ESkuNshQSuI3vsKBocngXjBp1LvWPg4Z160DkYqnpuXr+UANX30UJ46JC+OaBQdzauxUKWLTjqGMr6CI7Dp2ipNzKn4bodZ3Myn5pCBGYFfUsd392LghVSCGBkczlDL7w+ecwYoRD5xbpwQoulpqeR7/oEHwu0oGtaRWDGAL8vPlkSTrDYiMY0iHc3dWqlbSsPKDuS95rrpGUZLzsvfCCMYLOXgoJPB4+g09yx8K110JJicPmFulPQxfKLTjDziO6f0iruX/c1IV2EUE8NWsj+YWmWM+xxtZm5dOyiT+tmga4uypaLVW1mqufHwx7dzSMHm2k/7nxRoeNoNOByIVWZxh/IepApNVUoK8379/eh9zCM/x9zmaPWjF2nR6u7bEqN9l5eUF4OCRGJMPixfDII/Df/zpsvSLTBSIRCRWROSJSKCJZInLnRfZ9RkS2iMgpEckQkWcqbc8UkdMiUmB7/eb8O6jeuf6hhjFZUXON7q2b8PR1nfh5y2He+nUH8/eWmH5I98Hjpzl4olgHIg9WsZqr1Qo//ggdDyRTOtrWJ/Thh8ZXBy2eZ7pABEwCSoDmQBIwWUS6VbOvAHcDIcANwKMicnulfW5WSjWyva5zVqVrInnHMZoH+7Fp/wl3VkPzQA8MbUeP1sF8lJLOD7tLTT+/qKJu/aL06NCGYERQMj/6JjK6dBYZ0bbmuIpM3Q4IRqYKRCISBIwBXlRKFSillgHzgHFV7a+UmqiUWqeUKlNK7QTmAkNcV+OaS95xhOz8IrJyi0z/IaKZj8UiZwcrKKDU5POL0rLyCfDxonPLxu6uilZfttFxJV/NIkUS6NrVLi/dQccEI7ONmosFypRSu+zKNgJXXepAMVbbGgp8UmnTdBGxAOuBZ5RSG6s5fgIwASAiIoKUlJTa1/4i3llr5NFQQEmplRkL13Cqva9Dr1FTBQUFDr8/M2mo9xdeXG7LeaewiOB3PIuUlP3urlaVFm85TVRjWL50Sa2Oa6i/uwqeeH9tZ87k1N//zvebmlNebqWszHh+ycqC8ePL2f50c8b+/e80njmz7hdRSpnmhRFIDlcqewBIqcGxr2AELT+7siFAABAI/A04DDS91LliY2OVI50uKVO9XvlVxTw/X7V7fr7q9I//qrWZeQ69Rm0kJye77dqu0JDvb3LKHhX13Hz1yeI97q5KtQrPlKp2f1ug3vplR62Pbci/O6U8+/6iopQyplif/4qKOrcPsFbV4bPfpU1zIpIiIqqa1zKgAKi8oH0wcOoS530Uo6/oJqXUmYpypdRypdRppVSRUuoN4DhGsHOpOesPcLyolP8b0bVeM+417b4hMTT2gY0m7mfcuO8E5Val3+MNTFV56S5WXhsubZpTSsVfbLutj8hbRDoqpXbbinsBWy9yzH3A88AwpdSl2ikU51Zrdolyq+KTxXvp2aYJ9wyORsSll9caGF9vCwNbevO/bUc4cbqUJgE+7q7SBdZlG/2ffSJ19pCGJDLywkmuFeX1ZarBCkqpQmA28KqIBInIEGAk8FVV+4tIEvA6cK1SKr3StkgRGSIiviLibxvaHQ4sd+5dnO+XLYfJzC3iz1e110FIc4ghrb0pKbOyYNMhd1elSmsz8+jYrBFNA93TB6o5R1WTXAMDjfL6MlUgsnkYo1/nKDADeEgptRVARIaKSIHdvq8BYcAau7lCH9u2NQYmA/nAAYzh3cOVUi4baqSU4uPFe4kJD+L6bi1cdVmtgYsOttChWSNmrzPfQAWrVbEu+7hulmuAKk9yjYoyfq6cHqguzDZqDqVUHjCqmm1LgUZ2P8dc5DxbgZ4Or2AtLN+Ty+YDJ/jX6B54WfTTkOYYIsLovq2Z+MtOsnILiQoLcneVzkrPKeDE6VIdiBqoqvLSOYIZn4gajMmL99CssR+39m3t7qpoDcyo3q0RgdnrDri7KudZm2n0D+lApNWGDkROsmn/cZbvyWX8lTH4eXu5uzpaA9OqaQCD24cxe/1+U+WfS8vKJzTIl5hw8zylaeanA5GTfLx4L439vblzoAOGlGhaFUb3acO+vNOsNVGWjrSsfPpGhuiBOVqt6EDkBD9tPMB/Nx/muq7NaexvvuG1WsNwQ/cWBPp6mWbQQl5hCek5hbpZTqs1HYgcLC0rnye+NbIIzd90SOeU05wmyM+bG7q3YP6mQxSXlru7OqzL0v1DWt3oQORgy3Yfo9xqtNmXlZs7MaXm+cb0bcOp4jKe+X6j2//oWZuVj4+X6GVOtFrTgcjRbG3jFgEfb4teBE9zKl9v47/wTxsPuT2r++KdRwlv5MfWgyfdVgfNM+lA5GCb9h0nLMiXJ6+N1TnlNKdbnZF3NmeVO5eGWLk3h+2HT3H4RLHbA6LmeXQgcqC8whIW7zrG2Lg2PHp1Rx2ENKcb1C7s7FORiLjtCXzqsgzAM9ZK0sxHByIHWrD5EGVWxcjeegKr5hpxUSF888Ag2oYEEBrkQ5+2rk80WlJmZUP2cUTASzdJa3VguhQ/nmzu+gPENm9EF70qpeZCcVEhPH19J/4ycwMr9uZyZcdwl15/9rr95BaW8OKILhSXWhnULky3Bmi1ogORg+zLK2JtVj7PXN9JT+bTXO76bi1oEuDDzDXZLg1EZeVWPkoxljm5b0iMfu9rdaKb5hxk3saDAIzs3crNNdEuR/4+XtzapzW/bT1CXmGJy67706aDZOcV8WhCBx2EtDrTgcgBlFL8uP4A/aNDaBMSeOkDNM0Jbh/QlpJyq8syLVitig8X7aFzi8Zc06W5S66pNUw6EDnAtkMn2X20gFv0IAXNjTq3CKZ326bMXLPPJYlQf95ymL3HCnn06g5Y9DInWj3oQOQAczccxNsi3NSjpburol3m7hjQlj1HC5w+j0cpxQeLdtMuIojh3fX7XqsfHYjqqdyqmLfhIFfFRhAapJdG1txrRM9WBPl6MXPNPqde5/ftR9lx+BSPxHfQiz5q9Wa6QCQioSIyR0QKRSRLRO68yL4vi0ip3TLhBSLSzm57bxFJE5Ei29fejq7vqoxcDp8sZmQf3SynuV+Qnze39G7F/E0HOVlc6pRrpGXm8Y+5W2jW2Jdb9OAczQFMF4iASUAJ0BxIAiaLSLeL7P+tUqqR3SsdQER8gbnA10AI8AUw11buMPM2HCTI14trdWetZhJ/7B9JcamVeRsOOvzcaVn53PHpKg6fKCa/qJRN+084/Bra5cdUgUhEgoAxwItKqQKl1DJgHjCuDqeLx5gn9a5S6oxS6n1AgKsdVd/U9BzmrD9Av+gQAnz1KqyaOfRq04TOLRozc022w8+dmp5LSbkVMEbN6VQ+miOYbUJrLFCmlNplV7YRuOoix9wsInnAIeBDpdRkW3k3YJM6f/jQJlv5L5VPIiITgAkAERERpKSkXLSie/LL+dfqYsoULN+dw9Q5v9MhxDOCUUFBwSXvz5Pp+4O4kFKmby9h/Ee/ckVLb4e9N73zy85+7yXgdzyLlBTHDRfXv7vLk9kCUSOgcg75E0B1OXNmAVOAI8BA4AcROa6UmmE7V+V2g2rPpZSaYjsXnTp1UvHx8Ret6JZFuymzxUsFnGkaRXx8h4seYxYpKSlc6v48mb4/sOw8xvTtq/k9u4zlh6wOywQfkJ4Lq1MZ1bsV466IdngqH/27uzy5NBCJSArVP90sBx4DgiuVBwOnqjpAKbXN7scVIvIeMBaYARTU5ly1ZVv7Tq87pJnS5oPn/garyIbtiKCxZPcxvCzCq6O6E+zvU69zWa1WcnJyOH78OOXlxgqzTZo0Yfv27fWup1l58v15eXnRtGlTwsPDsVgc26vj0kCklIq/2HZbH5G3iHRUSu22FfcCttb0EnB2eZatwFMiInbNcz0xBkPUS7lVMW/jQdqEBHB7/7Zc0T5cJ3nUTGVQuzB8vSyUlFuxWBy3PMTS3Tn0adu03kEIYP/+/YgI0dHR+Pj4ICKcOnWKxo0bbtJgT70/pRSlpaUcOXKE/fv3ExkZ6dDzm2qwglKqEJgNvCoiQSIyBBgJfFXV/iIyUkRCxDAAeBxjpBxAClAOPC4ifiLyqK18UX3rOX/TQfYcLeBvw7vodYc0U4qLCmH6AwNp7OdN15bBDnmP5hWWsPnACYbFRjighlBYWEjr1q3x9fXVeepMTkTw9fWldevWFBYWOvz8pgpENg8DAcBRjCa2h5RSWwFEZKiIFNjtezuwB6O57UvgTaXUFwBKqRJgFHA3cBy4DxhlK6+zsnIr7y7cTecWjRnevUV9TqVpTtU/OpQ/DYlm04ETHDx+ut7nW7YnB6VgqAOzezu6iUdzLmf9vsw2WAGlVB5GAKlq21KMQQgVP99xiXOtB+IcWb8fNxwkI6eQT8bF6fxamumNjWvL+4v2MHvdfh69umO9zrVk1zGaBPjQs43rF9/TGjb950gtlJZbef/33XRrFcx1XfUEVs38IsMCGRgTyvdp++uVCFUpxdLdx7iyQ7hO6aM5nA5EtTB73X6y84p48tpY3aateYzb+rUlM7eINZl1T4S660gBR06eYVisa1d/1Wrn5Zdfpnv37u6uRq3pQFRDJWVW3v99D73aNuXqzs3cXR1Nq7Ebe7QgyNeL79bWPRHqkl3HABja0TEDFRqCdevW4eXlxZAhQ2p1XHx8PI8++uild7yM6EBUQ2//tpMDx09zS6+W+mlI8yiBvt6M6NmKBZsPUXim7NIHVGHJ7mN0aNaIVk0DHFy7+ps+HaKjwWIxvk6f7prrTp06lYcffpgtW7Z47Nwgs9CBqAZWpecyZUk6AG/9utPpa71omqPd1q8NRSXlLNh8qNbHFpeWszojj2EmfBqaPh0mTICsLFDK+DphgvOD0enTp/nmm2+YMGECY8eOZdq0aedtT01N5eqrryYoKIgmTZpw9dVXc/DgQf785z+zePFiJk2ahIggImRmZpKSkoKIkJOTc/YcmZmZiAhr164FoLy8nPHjxxMTE0NAQAAdO3Zk4sSJWK1W596sC5hu1JwZfb4i8+z3jpylrmmuEhcVQrvwIL5fu5/Efm1rdezqjDzOlFkZ6oL+oeee82PbtkvvVyE1Fc6cOb+sqAjGj4dPP63ZOXr3hnffrfk1Ab7//nuioqLo0aMH48aNIzExkTfeeAMfHx82btxIQkIC48aN45133sHPz48lS5ZQVlbGm2++SUZGBp07d+b1118HjNyWmZmZl7ym1WqldevWzJo1i4iICFavXs2ECRMICwtj/PjxtbsBk9GB6BKsVsWm/ccRdDofzXOJCGP7tWHiLzvJzCkkOjyoxscu2XUMXy8Lg2LM976vHIQuVe4o06ZNY9w4Y1GAq666isDAQObOncvYsWOZOHEivXv3ZsqUKWf379KlC2BkVvD19SUwMJAWLWo3D9HHx4dXX3317M/R0dGsW7eOGTNm6EDU0C3cfoQDx4t54pqO+HgZQUg/DWmeaEzfNrz1y06en72JZ67vXOP38dLdOfSPcc1SJ2++eYbGjWu+ZFh0tNEcV1lUFDgryfWePXtYtmwZ33zzDWAE+aSkJKZNm8bYsWNZv349t956q1Ou/fHHHzN16lSysrI4ffo0paWlREVFOeVarqQD0UUopfh48V7ahgbwaEIHvL10l5rmufbnn0YEUtPzSJqaWqOM3IdPFLPzyClG9+3solrWzj//afQJFRWdKwsMNMqdZerUqZSXl5+Xb61ijta+fXUbmViRscB+rldp6fkr7H777bc88cQTvP322wwePJjg4GAmTZrEnDlz6nRNM9GB6CLWZOazLvs4r47spoOQ5vFS03Op+Jw7U1qzvs6lu809bDspyfj6wguQnQ2RkUYQqih3tLKyMr744gveeOMNRowYcd62cePG8dlnn9GnTx8WLao+paWvr+/ZbOMVIiKMf99Dhw6d/X7Dhg3n7bNs2TIGDhx43tDvvXv31ut+zEIHoov4ePFeQoN8uS2udp27mmZGg9qF4edjobjUigKiwgIvecyS3TmEN/KjS0vzZoxOSnJe4KlswYIF5OTk8MADDxAWdn6f2e23387HH3/M7NmzueKKK5gwYQKPPPII/v7+LF26lOuuu46QKz/aBgAAEiZJREFUkBCio6NZvXo1mZmZNGrUiNDQUDp06EDbtm15+eWX+de//kVmZiavvfbaeeePjY3l888/5+eff6ZDhw7MnDmTxYsXExLi+V0F+s/8auw8fIpFO45y7+BovQy41iDERYUw/f5BPBTfnkBfL75ckYXVWn3aH6tVkbLjCM0b+7Eu+7gLa2pe06ZNIyEh4YIgBHDbbbeRmZnJsWPHWLhwITt27GDQoEEMHDiQmTNn4uNjLJ3x9NNP4+vrS9euXYmIiCA7OxsfHx9mzpxJeno6vXr14qWXXjo7qq7Cgw8+SGJiInfeeSf9+/cnMzOTp556yiX37WxSn/xTDVWnTp3UTa9+yy9bD7Pi+atpGljzzlNP0NBXidT3d2nfrd3HM99v4tWR3bj7iugq9/nngm18ujQDAfx8LA5b5bXC9u3bz44mq+Cp6/XUVEO4v6p+bxVEJE0p1a+259RPRFUos8LcjQe5Y0BkgwtCmgYwNq4Nw2Ij+NfPO9iXV3TeNqUU7y7cxadLM4yfOTd/TtOcQQeiKpwsUQgw/soYd1dF05xCRHhjdA8sIjw/e9PZ0Vpl5Vb+Pmcz7y7cTXxsBP4+Frz0/DnNyfRghSqcLFHc1CHclHm1NM1RWjcN4G83duaFOVv4ds0+RvZuzWMz1rNw+xEeSWjP09d1Yl32cVLTc/X8Oc2pdCCqxsr0XNKy8vV/Pq1Bu6N/JPM3HuLVn7bx3u+7OXSi+Lx+o7ioEP1/QHM60zXNiUioiMwRkUIRyRKROy+y788iUmD3KhGRzXbbM0XktN3232paj7Jy3SauNXwWi3DXoEiKSss5dKIYHy+hW6sm7q6Wdpkx4xPRJKAEaA70BhaIyEal1NbKOyqlhtv/LCIpQOWZZDcrpRbWthK6TVy7XGTmFmERsCpjyLZO6qu5mqmeiEQkCBgDvKiUKlD/v727D46qXg84/n2WhIRXkZeEm1wvMUK4AeQlKmp5EUccodHS1le0XKlibHgZ0XpHZ1BAcUAcXuK9FSvIlQh3uHW8lFYtdkZIykvF4UWsDVEUaawhaa8WQiBAuPD0j3N22SwBks1uzp71+cycIfv7nXN4nuxunt1zfud3VLcD/wxMacG2OcAY4O22xnFlmsR8qKoxieqm3F50TLFBCcY7ifaNKA/4o6oeCGv7DLilBdv+Atimqv8V0f5bEQkAnwK/VNXPLrejK9LEipD50Qhe6GqDEoxXEq0QdQWORbTVAS25AuwXwEsRbQ8BewEBngD+VUR+rqoXXCYuIkVAETjzPpXHa+reBHD8+HHLz8fild9ggfpD31F+KOa7btYVV1xBfX19k7azZ89e0JZMkiG/U6dOxf71p6rttgDlONfHNbdsB0YADRHb/C3w3mX2Oxo4DnS9zHpf4JwzumSceXl5mszKysq8DiGuLD9/2L9//wVtx44da9nGixerbtly6XW2bHHWSyAtzi+BNfe8BQG7NYra0K7niFR1nKrKRZbRwAEgRUQGhG02DLhgoEKEh4ENqnr8ciHgfDsyxvjZDTfAffdBWVnz/WVlTv8NN8Tlv586dWroVt/hS+SM2X4iIrz77rue/N8JNVhBVU8AG4AXRaSLiIwCJgFrL7aNiHQC7gPWRLT/TERGiUhHEUkXkV8CvYEdcUvAGNM+br0V3nmn+WIULELvvOOsFyfjx4+npqamyTJkyJBW76exsTEO0flLQhUi13SgE/C/wHqgWN2h2yIyRkQiv/X8OXAUiPxo1A14HTgCVAMTgImqahcHGZMMmitG7VSEANLS0ujbt2+TJSUlha1bt3LjjTeSnp5OZmYmTz75ZJNiM27cOIqLi3n66afp06cPo0aNAmD//v0UFhbSrVs3MjIymDx5MrW1tU3+z9LSUq699lrS0tLIzMzk4YcfDvUtW7aMoUOH0qVLF7Kzs5k2bRpHj54/HV5XV8eUKVPIyMggPT2d3NxcSkpKAOe24+DMIC4iocftJdEGK6Cq/4dTXJrr24YzoCG8bT1OwYpctwIYGo8YjTHxkfbMM7B/f+s2ysqCO+6An/wEamogPx9eeMFZWmL4cHD/ILdVdXU1EydOZMqUKaxZs4aDBw8ybdo0AoEAS5cuDa23bt06ioqK2LZtG6pKTU0NY8eO5dFHH2XJkiWcOXOGOXPmMGnSJD7++GMCgQBvvPEGTzzxBAsXLqSwsJDjx483uQFfIBCgpKSE3NxcqqqqmDVrFrNmzWLtWueA0nPPPcfnn3/O+++/T2ZmJocOHeIPf3BufLhr1y4yMjJYtWoVd955Jx06tO+tbxKuEBljTKtceaVThIK3aG2nG8V9+OGHdO16/nPxmDFjKCgoICsrixUrVhAIBMjPz+fll1/m8ccfZ8GCBaF1r7766iaFae7cuQwbNozFixeH2t5++2169uzJ7t27GTlyJAsWLGD27Nk89dRToXWuu+660M+zZ88O/ZyTk8Mrr7zCpEmTKC0tJRAIUFVVRUFBASNHjgSgX79+ofWDd4Xt0aMHffv2jcWvp1WsEBljEsbpxYvp2Nr79QQPxz3/PLz+OsybF/fDcgBjx45l5cqVocedOnVi1qxZ3HTTTQQC5896jB49msbGRr7++muuvtqZ0T+8gADs2bOHrVu3NilsQQcPHiQnJ4fq6mpuu+22i8azZcsWFi1aRGVlJXV1dZw9e5bGxkZqa2vJysqiuLiYe+65hz179nD77bdz1113ccstLblEM/6sEBlj/CvynNCtt7bbOaLOnTvTv3//Fq8vcn7AbpcuXZr0nTt3jsLCQpYsWXLBdpmZmZw8efKS+66qqqKwsJDHHnuMF198kV69erF3714mT54cOj81ceJEqqqq2LRpE5s3b6awsJB7772Xt956q8U5xEsiDlYwxpjLa25gwqVG07WD/Px8du7cyblz50Jt27dvp2PHjlxzzTUX3a6goICKigr69etH//79myzBwQvZ2dls3ry52e13795NY2Mjy5cv5+abbyYvL4/Dhw9fsF7v3r1D569Wr15NaWkpp0+fBiA1NZWzZ8+28TcQHStExhj/udToOA+L0fTp0zl8+DDTp0+nsrKSDz74gGeffZaZM2fSuXPni243Y8YM6urquP/++/nkk0/45ptv+OijjygqKgrNxDBnzhxKSkpYvnw5Bw4cYN++faHzTAMGDODcuXOUlJRw6NAh1q9fHxoRFzR37lw2btzIV199RWVlJRs2bCA3N5e0tDTAOa+0efNmamtrOXLkSJx+Q82zQmSM8Z9duy59+C1YjHbtatewsrOz2bRpE59++inDhw/nkUceYfLkySxcuPCS22VlZbFjxw4CgQATJkxg8ODBzJgxg7S0tFChKC4u5rXXXmPVqlUMGTKECRMmUFHhXOs/dOhQXn31VZYtW8agQYN48803LzjMl5aWxpw5cxg2bBijRo2ivr6e9957L9S/dOlSysrKuOqqqxgxYkSMfzOXJureIticN3DgQP3yyy+9DiNuysvLGTdunNdhxI3l5w+VlZXk5+c3aauvr6dbawcr+Egy5Nfc8xYkIntU9frW7tO+ERljjPGUFSJjjDGeskJkjDHGU1aIjDHGeMoKkTHGMzZYyl/i9XxZITLGeCI1NfWyMwaYxHLy5ElSU1Njvl8rRMYYT2RkZFBdXU1DQ4N9M0pwqkpDQwPV1dVkZGTEfP8215wxxhPdu3cH4PDhw5w5cwaAU6dOkZ6e7mVYceXn/FJTU8nMzAw9b7FkhcgY45nu3bs3+cNWXl7e7lf1t6dkzy9admjOGGOMp6wQGWOM8VTCFSIRmSkiu0XktIisacH6T4pIrYgcE5HfiEhaWF+OiJSJSIOIfCEi4+MavDHGmFZLuEIEHAZeAn5zuRVF5A7gWeA2oB+QC4TfqH498CnQC5gDvCsifWIdsDHGmOglXCFS1Q2quhH4oQWrPwysVtUKVT0CLACmAohIHlAAzFPVk6r6e+Bz4O74RG6MMSYafh81Nxj4p7DHnwGZItLL7ftGVesj+gc3tyMRKQKK3IenReQ/4xBvougNfO91EHFk+flXMucGyZ/fwGg28nsh6grUhT0O/tytmb5gf3ZzO1LVlcBKABHZHc09NfzC8vO3ZM4vmXODH0d+0WzXrofmRKRcRPQiy/YodnkcCL+6KvhzfTN9wf56jDHGJIx2LUSqOk5V5SLL6Ch2WQEMC3s8DPgfVf3B7csVkW4R/RXRZ2CMMSbWEm6wgoikiEg60AHoICLpInKxQ4hvA4+KyCAR6QE8B6wBUNUDwD5gnruPvwCGAr9vQRgr25pHgrP8/C2Z80vm3MDya5Yk2mSDIjIfmBfR/IKqzheRnwH7gUGq+q27/lPAM0AnnCLzN6p62u3LwSlMNwLfAjNU9aP4Z2GMMaalEq4QGWOM+XFJuENzxhhjflysEBljjPGUFaIwItJTRP5RRE6ISJWIPOh1TG1xqXn7ROQ2d/69Bnc+vn4ehRkVEUkTkdXu81QvIvtEZGJYv6/zAxCRdSJS486jeEBEpoX1+T4/ABEZICKnRGRdWNuD7vN6QkQ2ikhPL2OMlnu5yikROe4uX4b1+T5HEXlARCrdHA6KyBi3vdWvTStETb0GNAKZwEPA6yLS7EwMPtHsvH0i0hvYADwP9AR2A//Q7tG1TQrw38AtwBU4IybfcSe6TYb8ABYBOaraHfgz4CURuS6J8gPnPbcr+MB9v70BTMF5HzYAK7wJLSZmqmpXdxkIyZGjiNwOLAb+GmcCgbHAN9G+Nm2wgktEugBHgCHu0G9EZC1QrarPehpcG4nIS8BPVXWq+7gImKqqf+I+7oIz7cgIVf3Cs0DbSET+A2fS214kWX4iMhAoB54AepAE+YnIA8Bf4oyE7a+qfyUiC3GK74PuOtcAlUCviOm6Ep6IlAPrVPXNiHbf5ygi/44zz+fqiPao/rbYN6Lz8oA/BouQ66Jz0/ncYJzcAFDVE8BBfJyriGTiPIcVJFF+IrJCRBqAL4Aa4F9IgvxEpDvwIvBURFdkbgdxjlLktV90MbVIRL4XkR0iMs5t83WOItIBuB7oIyJfi8h3IvJ3ItKJKF+bVojO6woci2irw/namWwuNg+fL3MVkVTgt0Cp+6krafJT1ek4cY/BOeRxmuTIbwHOJ+rvItqTIbegZ3BuTZONc6Hne+63H7/nmAmkAvfgvC6HAyNwDo9HlZsVovN+THPTJU2uIhIA1uJ8opzpNidNfgCqelZVtwM/BYrxeX4iMhwYDyxvptvXuYVT1U9UtV5VT6tqKbAD+FP8n+NJ999fq2qNqn4PLKMNuVkhOu8AkCIiA8LaknVuuiZz9LnHca/BZ7mKiACrcT6h3a2qZ9yupMivGSmcz8PP+Y0DcoBvRaQWeBq4W0T2cmFuuUAazvvT7xQQfJ6je++373DyCTW7/0b32lRVW9wF+B3OXV27AKNwvlIO9jquNuSTAqTjjL5a6/6cAvRxc7vbbVsM7PQ63ijy+3tgJ9A1ot33+QEZwAM4hzo6AHcAJ3BGz/k6P6Az0DdsWQK86+Y1GOcQ+Rj3fbgO+J3XMUeRYw/3OQu+5x5yn7+8ZMgR5/zeLvd1eiWwDedwa1SvTc8TSqQFZ7jhRvcF8y3woNcxtTGf+TifVMKX+W7feJwT4CdxRmPleB1vK3Pr5+ZzCudwQHB5KEny6wP8G3DU/aP1OfBYWL+v84vIdT7O6LLg4wfd998JnBtf9vQ6xiifv104h6SO4nxguj1ZcsQ5R7TCza0W+BWQ7va1+rVpw7eNMcZ4ys4RGWOM8ZQVImOMMZ6yQmSMMcZTVoiMMcZ4ygqRMcYYT1khMsYY4ykrRMYYYzxlhcgYnxCR7iIyX0TyvY7FmFiyQmSMf1wPzMO5qt2YpGGFyBj/GIFzG4j9XgdiTCzZFD/G+ICIVAI/j2jeoKp3exGPMbFkhcgYHxCRG3Bmh68AFrrNNapa5V1UxsRGitcBGGNa5DOcG+P9WlV3eh2MMbFk54iM8YfBQEdgr9eBGBNrVoiM8YcCnPsv7fM6EGNizQqRMf4wAjioqse8DsSYWLNCZIw/DMKGbZskZYMVjPGHo0CBiNwB1AFfqeoPHsdkTEzY8G1jfEBEhgCrgaFAOjBGVbd7G5UxsWGFyBhjjKfsHJExxhhPWSEyxhjjKStExhhjPGWFyBhjjKesEBljjPGUFSJjjDGeskJkjDHGU1aIjDHGeOr/AUeQOFGtVK1AAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" } } ], "source": [ "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "V5kAzvP15jlD" }, "source": [ "## GRUs" ] }, { "cell_type": "code", "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.GRU(20, return_sequences=True, input_shape=[None, 1]),\n", " keras.layers.GRU(20, return_sequences=True),\n", " keras.layers.TimeDistributed(keras.layers.Dense(10))\n", "])\n", "\n", "model.summary()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "U0pp1tbVcFEP", "outputId": "3dedff16-df49-459d-aa13-4f607d010e1a" }, "execution_count": 57, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_8\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " gru (GRU) (None, None, 20) 1380 \n", " \n", " gru_1 (GRU) (None, None, 20) 2520 \n", " \n", " time_distributed_3 (TimeDis (None, None, 10) 210 \n", " tributed) \n", " \n", "=================================================================\n", "Total params: 4,110\n", "Trainable params: 4,110\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "id": "cV-aoeD95jlD", "outputId": "aa2fe19b-4891-4f77-e507-0751f2053dde", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 11s 31ms/step - loss: 0.0738 - last_time_step_mse: 0.0655 - val_loss: 0.0538 - val_last_time_step_mse: 0.0450\n", "Epoch 2/20\n", "219/219 [==============================] - 6s 29ms/step - loss: 0.0476 - last_time_step_mse: 0.0367 - val_loss: 0.0441 - val_last_time_step_mse: 0.0326\n", "Epoch 3/20\n", "219/219 [==============================] - 5s 25ms/step - loss: 0.0417 - last_time_step_mse: 0.0301 - val_loss: 0.0390 - val_last_time_step_mse: 0.0275\n", "Epoch 4/20\n", "219/219 [==============================] - 6s 27ms/step - loss: 0.0368 - last_time_step_mse: 0.0243 - val_loss: 0.0339 - val_last_time_step_mse: 0.0202\n", "Epoch 5/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0326 - last_time_step_mse: 0.0180 - val_loss: 0.0312 - val_last_time_step_mse: 0.0164\n", "Epoch 6/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0306 - last_time_step_mse: 0.0155 - val_loss: 0.0294 - val_last_time_step_mse: 0.0143\n", "Epoch 7/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0295 - last_time_step_mse: 0.0145 - val_loss: 0.0300 - val_last_time_step_mse: 0.0162\n", "Epoch 8/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0283 - last_time_step_mse: 0.0135 - val_loss: 0.0278 - val_last_time_step_mse: 0.0130\n", "Epoch 9/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0276 - last_time_step_mse: 0.0130 - val_loss: 0.0273 - val_last_time_step_mse: 0.0127\n", "Epoch 10/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0269 - last_time_step_mse: 0.0125 - val_loss: 0.0264 - val_last_time_step_mse: 0.0121\n", "Epoch 11/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0265 - last_time_step_mse: 0.0121 - val_loss: 0.0268 - val_last_time_step_mse: 0.0135\n", "Epoch 12/20\n", "219/219 [==============================] - 5s 25ms/step - loss: 0.0263 - last_time_step_mse: 0.0123 - val_loss: 0.0261 - val_last_time_step_mse: 0.0123\n", "Epoch 13/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0258 - last_time_step_mse: 0.0116 - val_loss: 0.0254 - val_last_time_step_mse: 0.0116\n", "Epoch 14/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0256 - last_time_step_mse: 0.0117 - val_loss: 0.0254 - val_last_time_step_mse: 0.0116\n", "Epoch 15/20\n", "219/219 [==============================] - 5s 24ms/step - loss: 0.0253 - last_time_step_mse: 0.0114 - val_loss: 0.0250 - val_last_time_step_mse: 0.0112\n", "Epoch 16/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0251 - last_time_step_mse: 0.0114 - val_loss: 0.0250 - val_last_time_step_mse: 0.0114\n", "Epoch 17/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0248 - last_time_step_mse: 0.0112 - val_loss: 0.0249 - val_last_time_step_mse: 0.0118\n", "Epoch 18/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0245 - last_time_step_mse: 0.0110 - val_loss: 0.0244 - val_last_time_step_mse: 0.0108\n", "Epoch 19/20\n", "219/219 [==============================] - 5s 23ms/step - loss: 0.0243 - last_time_step_mse: 0.0108 - val_loss: 0.0240 - val_last_time_step_mse: 0.0105\n", "Epoch 20/20\n", "219/219 [==============================] - 5s 25ms/step - loss: 0.0240 - last_time_step_mse: 0.0106 - val_loss: 0.0238 - val_last_time_step_mse: 0.0103\n" ] } ], "source": [ "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n", "history = model.fit(X_train, Y_train, epochs=20,\n", " validation_data=(X_valid, Y_valid))" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "id": "TNOeSZW_5jlD", "outputId": "7e827299-42da-4e52-e341-3ed81d0333e9", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 1s 9ms/step - loss: 0.0238 - last_time_step_mse: 0.0103\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "[0.02378549985587597, 0.010262805968523026]" ] }, "metadata": {}, "execution_count": 58 } ], "source": [ "model.evaluate(X_valid, Y_valid)" ] }, { "cell_type": "markdown", "metadata": { "id": "9gDnHqvg5jlD" }, "source": [ "## Using One-Dimensional Convolutional Layers to Process Sequences" ] }, { "cell_type": "markdown", "source": [ "The following model is the same as earlier, except it starts with a 1D convolutional layer that downsamples the input sequence by a factor of 2, using\n", "a stride of 2. By shortening the sequences, the convolutional layer may help the GRU layers detect longer patterns. Note that we must also crop off the first three time steps in the targets (since the kernel’s size is 4, the first output of the convolutional layer will be based on the input time steps 0 to 3), and downsample the targets by a factor of 2:" ], "metadata": { "id": "yTdnZoTLYjGN" } }, { "cell_type": "code", "execution_count": 59, "metadata": { "id": "9PKOidly5jlE", "outputId": "d77c8947-1acc-497e-b7b0-df3dcbfb0fe9", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"sequential_9\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " conv1d (Conv1D) (None, None, 20) 100 \n", " \n", " gru_2 (GRU) (None, None, 20) 2520 \n", " \n", " gru_3 (GRU) (None, None, 20) 2520 \n", " \n", " time_distributed_4 (TimeDis (None, None, 10) 210 \n", " tributed) \n", " \n", "=================================================================\n", "Total params: 5,350\n", "Trainable params: 5,350\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.Conv1D(filters=20, kernel_size=4, strides=2, padding=\"valid\",\n", " input_shape=[None, 1]),\n", " keras.layers.GRU(20, return_sequences=True),\n", " keras.layers.GRU(20, return_sequences=True),\n", " keras.layers.TimeDistributed(keras.layers.Dense(10))\n", "])\n", "\n", "model.summary()" ] }, { "cell_type": "code", "source": [ "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n", "history = model.fit(X_train, Y_train[:, 3::2], epochs=20,\n", " validation_data=(X_valid, Y_valid[:, 3::2]))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "kMugaQpAcKFB", "outputId": "2b4564bd-988a-4735-b9a2-c6e029300045" }, "execution_count": 60, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/20\n", "219/219 [==============================] - 13s 19ms/step - loss: 0.0681 - last_time_step_mse: 0.0601 - val_loss: 0.0477 - val_last_time_step_mse: 0.0396\n", "Epoch 2/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0414 - last_time_step_mse: 0.0340 - val_loss: 0.0367 - val_last_time_step_mse: 0.0285\n", "Epoch 3/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0338 - last_time_step_mse: 0.0257 - val_loss: 0.0307 - val_last_time_step_mse: 0.0218\n", "Epoch 4/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0282 - last_time_step_mse: 0.0184 - val_loss: 0.0259 - val_last_time_step_mse: 0.0152\n", "Epoch 5/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0249 - last_time_step_mse: 0.0143 - val_loss: 0.0246 - val_last_time_step_mse: 0.0141\n", "Epoch 6/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0234 - last_time_step_mse: 0.0125 - val_loss: 0.0227 - val_last_time_step_mse: 0.0115\n", "Epoch 7/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0226 - last_time_step_mse: 0.0117 - val_loss: 0.0225 - val_last_time_step_mse: 0.0116\n", "Epoch 8/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0220 - last_time_step_mse: 0.0111 - val_loss: 0.0216 - val_last_time_step_mse: 0.0105\n", "Epoch 9/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0216 - last_time_step_mse: 0.0108 - val_loss: 0.0217 - val_last_time_step_mse: 0.0109\n", "Epoch 10/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0213 - last_time_step_mse: 0.0106 - val_loss: 0.0210 - val_last_time_step_mse: 0.0102\n", "Epoch 11/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0210 - last_time_step_mse: 0.0102 - val_loss: 0.0208 - val_last_time_step_mse: 0.0100\n", "Epoch 12/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0208 - last_time_step_mse: 0.0102 - val_loss: 0.0208 - val_last_time_step_mse: 0.0102\n", "Epoch 13/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0205 - last_time_step_mse: 0.0098 - val_loss: 0.0206 - val_last_time_step_mse: 0.0101\n", "Epoch 14/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0204 - last_time_step_mse: 0.0099 - val_loss: 0.0204 - val_last_time_step_mse: 0.0099\n", "Epoch 15/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0202 - last_time_step_mse: 0.0097 - val_loss: 0.0199 - val_last_time_step_mse: 0.0093\n", "Epoch 16/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0200 - last_time_step_mse: 0.0097 - val_loss: 0.0201 - val_last_time_step_mse: 0.0095\n", "Epoch 17/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0196 - last_time_step_mse: 0.0093 - val_loss: 0.0197 - val_last_time_step_mse: 0.0091\n", "Epoch 18/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0194 - last_time_step_mse: 0.0090 - val_loss: 0.0192 - val_last_time_step_mse: 0.0086\n", "Epoch 19/20\n", "219/219 [==============================] - 4s 16ms/step - loss: 0.0190 - last_time_step_mse: 0.0087 - val_loss: 0.0188 - val_last_time_step_mse: 0.0084\n", "Epoch 20/20\n", "219/219 [==============================] - 3s 15ms/step - loss: 0.0186 - last_time_step_mse: 0.0083 - val_loss: 0.0184 - val_last_time_step_mse: 0.0080\n" ] } ] }, { "cell_type": "code", "source": [ "model.evaluate(X_valid, Y_valid[:, 3::2])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DlBLXViLcOxn", "outputId": "cf835442-e81f-4be9-8b09-22942cca6d01" }, "execution_count": 62, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "63/63 [==============================] - 1s 8ms/step - loss: 0.0184 - last_time_step_mse: 0.0080\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "[0.018366245552897453, 0.00796824786812067]" ] }, "metadata": {}, "execution_count": 62 } ] }, { "cell_type": "markdown", "source": [ "# Natural-language processing" ], "metadata": { "id": "e16QVbDxcPb2" } }, { "cell_type": "markdown", "source": [ "## Preparing text data" ], "metadata": { "id": "Vq_YGZBzcZsn" } }, { "cell_type": "markdown", "source": [ "Vectorizing process using Python may be done as follows" ], "metadata": { "id": "ACRmbWfKdbvv" } }, { "cell_type": "code", "source": [ "import string\n", "\n", "class Vectorizer:\n", " def standardize(self, text):\n", " text = text.lower()\n", " return \"\".join(char for char in text if char not in string.punctuation)\n", "\n", " def tokenize(self, text):\n", " text = self.standardize(text)\n", " return text.split()\n", "\n", " def make_vocabulary(self, dataset):\n", " self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n", " for text in dataset:\n", " text = self.standardize(text)\n", " tokens = self.tokenize(text)\n", " for token in tokens:\n", " if token not in self.vocabulary:\n", " self.vocabulary[token] = len(self.vocabulary)\n", " self.inverse_vocabulary = dict(\n", " (v, k) for k, v in self.vocabulary.items())\n", "\n", " def encode(self, text):\n", " text = self.standardize(text)\n", " tokens = self.tokenize(text)\n", " return [self.vocabulary.get(token, 1) for token in tokens]\n", "\n", " def decode(self, int_sequence):\n", " return \" \".join(\n", " self.inverse_vocabulary.get(i, \"[UNK]\") for i in int_sequence)\n", "\n", "vectorizer = Vectorizer()\n", "dataset = [\n", " \"I write, erase, rewrite\",\n", " \"Erase again, and then\",\n", " \"A poppy blooms.\",\n", "]\n", "vectorizer.make_vocabulary(dataset)" ], "metadata": { "id": "JPAOQHVAdZTR" }, "execution_count": 63, "outputs": [] }, { "cell_type": "code", "source": [ "test_sentence = \"I write, rewrite, and still rewrite again\"\n", "encoded_sentence = vectorizer.encode(test_sentence)\n", "print(encoded_sentence)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VGgm3yP3drce", "outputId": "6b368c1a-3ec3-48e9-e7f1-ad80eae4dc9c" }, "execution_count": 64, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[2, 3, 5, 7, 1, 5, 6]\n" ] } ] }, { "cell_type": "code", "source": [ "decoded_sentence = vectorizer.decode(encoded_sentence)\n", "print(decoded_sentence)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yeVLfOrAdxRf", "outputId": "7f564f9f-8911-46dd-c9bd-04f4ffde58a4" }, "execution_count": 65, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "i write rewrite and [UNK] rewrite again\n" ] } ] }, { "cell_type": "markdown", "source": [ "However, using something like this wouldn’t be very performant. In practice, you’ll work with the Keras `TextVectorization` layer, which is fast and efficient and can be dropped directly into a `tf.data` pipeline or a Keras model." ], "metadata": { "id": "cF6Xq7YMcip3" } }, { "cell_type": "code", "source": [ "from tensorflow.keras.layers import TextVectorization\n", "# Configures the layer to return sequences of words encoded\n", "# as integer indices.\n", "text_vectorization = TextVectorization(\n", " output_mode=\"int\",\n", ")" ], "metadata": { "id": "7oWACab-cTGJ" }, "execution_count": 66, "outputs": [] }, { "cell_type": "markdown", "source": [ "By default, the TextVectorization layer will use the setting “convert to lowercase and remove punctuation” for text standardization, and “split on whitespace” for tokenization.\n", "\n", "But importantly, you can provide custom functions for standardization and tokenization, which means the layer is flexible enough to handle any use case.To index the vocabulary of a text corpus, just call the `adapt()` method of the layer with a `Dataset` object that yields strings, or just with a list of Python strings:" ], "metadata": { "id": "t8PWeaj0eUSp" } }, { "cell_type": "code", "source": [ "dataset = [\n", " \"I write, erase, rewrite\",\n", " \"Erase again, and then\",\n", " \"A poppy blooms.\",\n", "]\n", "text_vectorization.adapt(dataset)" ], "metadata": { "id": "9IqhI0sceL0Q" }, "execution_count": 67, "outputs": [] }, { "cell_type": "markdown", "source": [ "Note that you can retrieve the computed vocabulary via `get_vocabulary()`—this can be useful if you need to convert text encoded as integer sequences back into words. The first two entries in the vocabulary are the mask token (index 0) and the OOV token (index 1). Entries in the vocabulary list are sorted by frequency, so with a realworld dataset, very common words like “the” or “a” would come first." ], "metadata": { "id": "tMljNvynex1J" } }, { "cell_type": "code", "source": [ "text_vectorization.get_vocabulary()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "y3XeB2HYegB6", "outputId": "3398c130-6067-4b33-9979-679a4d2eb66d" }, "execution_count": 68, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['',\n", " '[UNK]',\n", " 'erase',\n", " 'write',\n", " 'then',\n", " 'rewrite',\n", " 'poppy',\n", " 'i',\n", " 'blooms',\n", " 'and',\n", " 'again',\n", " 'a']" ] }, "metadata": {}, "execution_count": 68 } ] }, { "cell_type": "code", "source": [ "vocabulary = text_vectorization.get_vocabulary()\n", "test_sentence = \"I write, rewrite, and still rewrite again\"\n", "encoded_sentence = text_vectorization(test_sentence)\n", "print(encoded_sentence)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3OldsH4Qe4Qb", "outputId": "da56367e-1b43-4285-ebf7-36d48b1451a6" }, "execution_count": 69, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "tf.Tensor([ 7 3 5 9 1 5 10], shape=(7,), dtype=int64)\n" ] } ] }, { "cell_type": "code", "source": [ "inverse_vocab = dict(enumerate(vocabulary))\n", "decoded_sentence = \" \".join(inverse_vocab[int(i)] for i in encoded_sentence)\n", "print(decoded_sentence)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "v94pHFXUe-MO", "outputId": "b3a5e18d-79df-4594-8d6f-79f0933a4fa4" }, "execution_count": 70, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "i write rewrite and [UNK] rewrite again\n" ] } ] }, { "cell_type": "markdown", "source": [ "Before dive into the modeling part. We’ll demonstrate each approach on a well-known text classification benchmark: the IMDB movie review sentiment-classification dataset. Let’s process the raw\n", "IMDB text data, just like you would do when approaching a new text-classification problem in the real world. \n", "\n", "https://ai.stanford.edu/~amaas/data/sentiment/\n", "\n", "Let’s start by downloading the dataset from the Stanford page of Andrew Maas and uncompressing it" ], "metadata": { "id": "DVxg9nzSfPuY" } }, { "cell_type": "code", "source": [ "!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n", "!tar -xf aclImdb_v1.tar.gz" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WWTJRDZXfOtX", "outputId": "b342fa45-bee0-423a-936a-d0176d393283" }, "execution_count": 71, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "100 80.2M 100 80.2M 0 0 10.1M 0 0:00:07 0:00:07 --:--:-- 16.5M\n" ] } ] }, { "cell_type": "markdown", "source": [ "There’s also a `train/unsup` subdirectory in there, which we don’t need. Let’s\n", "delete it:" ], "metadata": { "id": "9Firy_UuhRf4" } }, { "cell_type": "code", "source": [ "!rm -r aclImdb/train/unsup" ], "metadata": { "id": "y5jDnHImg4lT" }, "execution_count": 72, "outputs": [] }, { "cell_type": "markdown", "source": [ "Take a look at the content of a few of these text files. Whether you’re working with text data or image data, remember to always inspect what your data looks like before you dive into modeling it." ], "metadata": { "id": "jdRkfu9uhXbx" } }, { "cell_type": "code", "source": [ "!cat aclImdb/train/pos/4077_10.txt" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DzJMWYO0hZOO", "outputId": "250da9ff-4194-402a-a1a4-d6d791c92d1e" }, "execution_count": 73, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "I first saw this back in the early 90s on UK TV, i did like it then but i missed the chance to tape it, many years passed but the film always stuck with me and i lost hope of seeing it TV again, the main thing that stuck with me was the end, the hole castle part really touched me, its easy to watch, has a great story, great music, the list goes on and on, its OK me saying how good it is but everyone will take there own best bits away with them once they have seen it, yes the animation is top notch and beautiful to watch, it does show its age in a very few parts but that has now become part of it beauty, i am so glad it has came out on DVD as it is one of my top 10 films of all time. Buy it or rent it just see it, best viewing is at night alone with drink and food in reach so you don't have to stop the film.

Enjoy" ] } ] }, { "cell_type": "markdown", "source": [ "Next, let’s prepare a validation set by setting apart 20% of the training text files in a new directory, aclImdb/val:" ], "metadata": { "id": "oEplxGV0hcyS" } }, { "cell_type": "code", "source": [ "import os, pathlib, shutil, random\n", "\n", "base_dir = pathlib.Path(\"aclImdb\")\n", "val_dir = base_dir / \"val\"\n", "train_dir = base_dir / \"train\"\n", "for category in (\"neg\", \"pos\"):\n", " os.makedirs(val_dir / category)\n", " files = os.listdir(train_dir / category)\n", " random.Random(1337).shuffle(files)\n", " num_val_samples = int(0.2 * len(files))\n", " val_files = files[-num_val_samples:]\n", " for fname in val_files:\n", " shutil.move(train_dir / category / fname,\n", " val_dir / category / fname)" ], "metadata": { "id": "uI1RHMXfheLP" }, "execution_count": 74, "outputs": [] }, { "cell_type": "markdown", "source": [ "Let’s create three Dataset objects for training, validation, and testing just like previous lab:" ], "metadata": { "id": "9rwl4rFphkdx" } }, { "cell_type": "code", "source": [ "from tensorflow import keras\n", "batch_size = 32\n", "\n", "train_ds = keras.utils.text_dataset_from_directory(\n", " \"aclImdb/train\", batch_size=batch_size\n", ")\n", "val_ds = keras.utils.text_dataset_from_directory(\n", " \"aclImdb/val\", batch_size=batch_size\n", ")\n", "test_ds = keras.utils.text_dataset_from_directory(\n", " \"aclImdb/test\", batch_size=batch_size\n", ")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "672YOyLXhnyI", "outputId": "63913c02-61eb-44ab-bff5-7e830afa81e2" }, "execution_count": 75, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Found 20000 files belonging to 2 classes.\n", "Found 5000 files belonging to 2 classes.\n", "Found 25000 files belonging to 2 classes.\n" ] } ] }, { "cell_type": "markdown", "source": [ "These datasets yield inputs that are TensorFlow `tf.string` tensors and targets that are `int32` tensors encoding the value “0” or “1.”" ], "metadata": { "id": "tILAl18fh9FA" } }, { "cell_type": "code", "source": [ "for inputs, targets in train_ds:\n", " print(\"inputs.shape:\", inputs.shape)\n", " print(\"inputs.dtype:\", inputs.dtype)\n", " print(\"targets.shape:\", targets.shape)\n", " print(\"targets.dtype:\", targets.dtype)\n", " print(\"inputs[0]:\", inputs[0])\n", " print(\"targets[0]:\", targets[0])\n", " break" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "gD_XsKfih3XR", "outputId": "9a9ceccd-17c1-4f8b-d128-6449ef5f28c0" }, "execution_count": 76, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "inputs.shape: (32,)\n", "inputs.dtype: \n", "targets.shape: (32,)\n", "targets.dtype: \n", "inputs[0]: tf.Tensor(b'There\\'s nothing really to dislike about \"The Odd Couple,\" and it\\'s no surprise that Jack Lemmon and Walter Matthau make a hugely winning comedic team. But there\\'s something so underdeveloped about Neil Simon\\'s adaptation of his hit stage play as to make it seem more like a skit on a sketch comedy show than a full-bodied film. I have not seen the play, but have to assume that the screen version is fairly faithful, since Simon wrote it, so the defects cannot be blamed on a stage-to-screen adaptation. There are some interesting ideas in this story--two recently divorced men who fall immediately into traditional married roles when they become roommates because neither knows any differently--that Simon never fully fleshes out. Still, there are many worse ways to kill a couple of hours.', shape=(), dtype=string)\n", "targets[0]: tf.Tensor(1, shape=(), dtype=int32)\n" ] } ] }, { "cell_type": "markdown", "source": [ "## Processing words as a set: The bag-of-words approach" ], "metadata": { "id": "nXU16HSgiGAK" } }, { "cell_type": "markdown", "source": [ "### Single words (unigrams) with binary encoding" ], "metadata": { "id": "6akoHLcIkVXr" } }, { "cell_type": "markdown", "source": [ "First, let’s process our raw text datasets with a `TextVectorization` layer so that they yield multi-hot encoded binary word vectors. Our layer will only look at single words (that is to say, unigrams). We will limit the vocabulary to the 20,000 most frequent words. Otherwise we’d be indexing every word in the training data— potentially tens of thousands of terms that only occur once or\n", "twice and thus aren’t informative. In general, 20,000 is the right vocabulary size for text classification." ], "metadata": { "id": "E0-NGwoeiNvx" } }, { "cell_type": "code", "source": [ "# Encode the output tokens as multi-hot binary vectors.\n", "text_vectorization = TextVectorization(\n", " max_tokens=20000,\n", " output_mode=\"multi_hot\",\n", ")\n", "# Prepare a dataset that only yields raw text inputs (no labels).\n", "text_only_train_ds = train_ds.map(lambda x, y: x)\n", "text_vectorization.adapt(text_only_train_ds)\n", "\n", "binary_1gram_train_ds = train_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "binary_1gram_val_ds = val_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "binary_1gram_test_ds = test_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)" ], "metadata": { "id": "SV_WnlbKh2xK" }, "execution_count": 77, "outputs": [] }, { "cell_type": "code", "source": [ "for inputs, targets in binary_1gram_train_ds:\n", " print(\"inputs.shape:\", inputs.shape)\n", " print(\"inputs.dtype:\", inputs.dtype)\n", " print(\"targets.shape:\", targets.shape)\n", " print(\"targets.dtype:\", targets.dtype)\n", " print(\"inputs[0]:\", inputs[0])\n", " print(\"targets[0]:\", targets[0])\n", " break" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rgJwHXctiuIH", "outputId": "b167570c-c6d3-4b6e-c0d3-d4ac00780454" }, "execution_count": 78, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "inputs.shape: (32, 20000)\n", "inputs.dtype: \n", "targets.shape: (32,)\n", "targets.dtype: \n", "inputs[0]: tf.Tensor([1. 1. 1. ... 0. 0. 0.], shape=(20000,), dtype=float32)\n", "targets[0]: tf.Tensor(0, shape=(), dtype=int32)\n" ] } ] }, { "cell_type": "code", "source": [ "from tensorflow import keras\n", "from tensorflow.keras import layers\n", "\n", "# A densely connected NN\n", "def get_model(max_tokens=20000, hidden_dim=16):\n", " inputs = keras.Input(shape=(max_tokens,))\n", " x = layers.Dense(hidden_dim, activation=\"relu\")(inputs)\n", " x = layers.Dropout(0.5)(x)\n", " outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n", " model = keras.Model(inputs, outputs)\n", " model.compile(optimizer=\"nadam\",\n", " loss=\"binary_crossentropy\",\n", " metrics=[\"accuracy\"])\n", " return model" ], "metadata": { "id": "DqwEhyqfiyma" }, "execution_count": 79, "outputs": [] }, { "cell_type": "markdown", "source": [ "Finally, let’s train and test our model." ], "metadata": { "id": "yMTMyNB5j0xq" } }, { "cell_type": "code", "source": [ "model = get_model()\n", "model.summary()\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"binary_1gram.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(binary_1gram_train_ds.cache(),\n", " validation_data=binary_1gram_val_ds.cache(),\n", " epochs=10,\n", " callbacks=callbacks)\n", "model = keras.models.load_model(\"binary_1gram.keras\")\n", "print(f\"Test acc: {model.evaluate(binary_1gram_test_ds)[1]:.3f}\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "h1ws10OIjt_X", "outputId": "eba30619-0f49-4880-d7e9-219acae9bc26" }, "execution_count": 80, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_1 (InputLayer) [(None, 20000)] 0 \n", " \n", " dense_8 (Dense) (None, 16) 320016 \n", " \n", " dropout (Dropout) (None, 16) 0 \n", " \n", " dense_9 (Dense) (None, 1) 17 \n", " \n", "=================================================================\n", "Total params: 320,033\n", "Trainable params: 320,033\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/10\n", "625/625 [==============================] - 13s 16ms/step - loss: 0.3879 - accuracy: 0.8375 - val_loss: 0.2680 - val_accuracy: 0.8932\n", "Epoch 2/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.2204 - accuracy: 0.9155 - val_loss: 0.2628 - val_accuracy: 0.8892\n", "Epoch 3/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.1569 - accuracy: 0.9441 - val_loss: 0.2857 - val_accuracy: 0.8852\n", "Epoch 4/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.1172 - accuracy: 0.9590 - val_loss: 0.3216 - val_accuracy: 0.8860\n", "Epoch 5/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0866 - accuracy: 0.9699 - val_loss: 0.3537 - val_accuracy: 0.8838\n", "Epoch 6/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0733 - accuracy: 0.9723 - val_loss: 0.3830 - val_accuracy: 0.8850\n", "Epoch 7/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0600 - accuracy: 0.9764 - val_loss: 0.4238 - val_accuracy: 0.8848\n", "Epoch 8/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0557 - accuracy: 0.9786 - val_loss: 0.4402 - val_accuracy: 0.8814\n", "Epoch 9/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0472 - accuracy: 0.9808 - val_loss: 0.4760 - val_accuracy: 0.8854\n", "Epoch 10/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0446 - accuracy: 0.9818 - val_loss: 0.5129 - val_accuracy: 0.8838\n", "782/782 [==============================] - 11s 14ms/step - loss: 0.2819 - accuracy: 0.8843\n", "Test acc: 0.884\n" ] } ] }, { "cell_type": "markdown", "source": [ "This gets us to a test accuracy of 88.4%: not bad!" ], "metadata": { "id": "YbuIiJwYkGiR" } }, { "cell_type": "markdown", "source": [ "### Bigrams with binary encoding" ], "metadata": { "id": "2SoPSzSrkOXL" } }, { "cell_type": "markdown", "source": [ "The `TextVectorization` layer can be configured to return arbitrary N-grams: bigrams, trigrams, etc. Just pass an `ngrams=N` argument as in the following listing." ], "metadata": { "id": "YOwpIaJjkeka" } }, { "cell_type": "code", "source": [ "text_vectorization = TextVectorization(\n", " ngrams=2,\n", " max_tokens=20000,\n", " output_mode=\"multi_hot\",\n", ")" ], "metadata": { "id": "GC3ZrIQmj3VB" }, "execution_count": 81, "outputs": [] }, { "cell_type": "code", "source": [ "text_vectorization.adapt(text_only_train_ds)\n", "binary_2gram_train_ds = train_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "binary_2gram_val_ds = val_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "binary_2gram_test_ds = test_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "\n", "model = get_model()\n", "model.summary()\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"binary_2gram.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(binary_2gram_train_ds.cache(),\n", " validation_data=binary_2gram_val_ds.cache(),\n", " epochs=10,\n", " callbacks=callbacks)\n", "model = keras.models.load_model(\"binary_2gram.keras\")\n", "print(f\"Test acc: {model.evaluate(binary_2gram_test_ds)[1]:.3f}\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SdEQkw3mkhk9", "outputId": "5cfc6f2d-3521-43e2-8c05-d995b9824d7c" }, "execution_count": 82, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_2 (InputLayer) [(None, 20000)] 0 \n", " \n", " dense_10 (Dense) (None, 16) 320016 \n", " \n", " dropout_1 (Dropout) (None, 16) 0 \n", " \n", " dense_11 (Dense) (None, 1) 17 \n", " \n", "=================================================================\n", "Total params: 320,033\n", "Trainable params: 320,033\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/10\n", "625/625 [==============================] - 24s 37ms/step - loss: 0.3597 - accuracy: 0.8476 - val_loss: 0.2467 - val_accuracy: 0.8980\n", "Epoch 2/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.1961 - accuracy: 0.9275 - val_loss: 0.2507 - val_accuracy: 0.8930\n", "Epoch 3/10\n", "625/625 [==============================] - 5s 8ms/step - loss: 0.1328 - accuracy: 0.9531 - val_loss: 0.2753 - val_accuracy: 0.8994\n", "Epoch 4/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0990 - accuracy: 0.9656 - val_loss: 0.3075 - val_accuracy: 0.8952\n", "Epoch 5/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0778 - accuracy: 0.9732 - val_loss: 0.3100 - val_accuracy: 0.8950\n", "Epoch 6/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0642 - accuracy: 0.9763 - val_loss: 0.3855 - val_accuracy: 0.8902\n", "Epoch 7/10\n", "625/625 [==============================] - 5s 7ms/step - loss: 0.0558 - accuracy: 0.9809 - val_loss: 0.3822 - val_accuracy: 0.8930\n", "Epoch 8/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0494 - accuracy: 0.9816 - val_loss: 0.4002 - val_accuracy: 0.8916\n", "Epoch 9/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0451 - accuracy: 0.9825 - val_loss: 0.4233 - val_accuracy: 0.8892\n", "Epoch 10/10\n", "625/625 [==============================] - 4s 6ms/step - loss: 0.0419 - accuracy: 0.9846 - val_loss: 0.4674 - val_accuracy: 0.8910\n", "782/782 [==============================] - 9s 11ms/step - loss: 0.2605 - accuracy: 0.8953\n", "Test acc: 0.895\n" ] } ] }, { "cell_type": "markdown", "source": [ "We’re now getting 89.5% test accuracy, a marked improvement! Turns out local order is pretty important." ], "metadata": { "id": "0aGRj_DRkqtD" } }, { "cell_type": "markdown", "source": [ "### Bigrams with TF-IDF encoding" ], "metadata": { "id": "CB3pEfxck15o" } }, { "cell_type": "markdown", "source": [ "TF-IDF is so common that it’s built into the TextVectorization layer. All you need to do to start using it is to switch the output_mode argument to `tf_idf`." ], "metadata": { "id": "tPG88cwWk5zS" } }, { "cell_type": "code", "source": [ "from keras.layers import TextVectorization as TexVec" ], "metadata": { "id": "uRaK89lhnenM" }, "execution_count": 91, "outputs": [] }, { "cell_type": "code", "source": [ "text_vectorization = TexVec(\n", " ngrams=2,\n", " max_tokens=20000,\n", " output_mode=\"tf_idf\",\n", ")\n", "\n", "text_vectorization.adapt(text_only_train_ds)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 503 }, "id": "m2V7i76UklOO", "outputId": "fd33c64c-57cd-44ba-829c-99ddcb61c660" }, "execution_count": 92, "outputs": [ { "output_type": "error", "ename": "InvalidArgumentError", "evalue": "ignored", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mInvalidArgumentError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m )\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mtext_vectorization\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madapt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext_only_train_ds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/layers/preprocessing/text_vectorization.py\u001b[0m in \u001b[0;36madapt\u001b[0;34m(self, data, batch_size, steps)\u001b[0m\n\u001b[1;32m 426\u001b[0m \u001b[0margument\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msupported\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0marray\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 427\u001b[0m \"\"\"\n\u001b[0;32m--> 428\u001b[0;31m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madapt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 429\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 430\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mupdate_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/engine/base_preprocessing_layer.py\u001b[0m in \u001b[0;36madapt\u001b[0;34m(self, data, batch_size, steps)\u001b[0m\n\u001b[1;32m 247\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcatch_stop_iteration\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 248\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msteps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 249\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_adapt_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 250\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 251\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 151\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 152\u001b[0m \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 153\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_tb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 154\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 155\u001b[0m \u001b[0;32mdel\u001b[0m \u001b[0mfiltered_tb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 53\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 54\u001b[0m tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 55\u001b[0;31m inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m 56\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mInvalidArgumentError\u001b[0m: Graph execution error:\n\n2 root error(s) found.\n (0) INVALID_ARGUMENT: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string\n\t [[{{node map/TensorArrayUnstack/TensorListFromTensor/_42}}]]\n\t [[add/_44]]\n (1) INVALID_ARGUMENT: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string\n\t [[{{node map/TensorArrayUnstack/TensorListFromTensor/_42}}]]\n0 successful operations.\n0 derived errors ignored. [Op:__inference_adapt_step_304531]" ] } ] }, { "cell_type": "code", "source": [ "text_vectorization.adapt(text_only_train_ds)\n", "\n", "tfidf_2gram_train_ds = train_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "tfidf_2gram_val_ds = val_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "tfidf_2gram_test_ds = test_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "\n", "model = get_model()\n", "model.summary()\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"tfidf_2gram.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(tfidf_2gram_train_ds.cache(),\n", " validation_data=tfidf_2gram_val_ds.cache(),\n", " epochs=10,\n", " callbacks=callbacks)\n", "model = keras.models.load_model(\"tfidf_2gram.keras\")\n", "print(f\"Test acc: {model.evaluate(tfidf_2gram_test_ds)[1]:.3f}\")" ], "metadata": { "id": "MDp4V3VjlGVh" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "This gets us an 89.8% test accuracy on the IMDB classification task: it doesn’t seem to be particularly helpful in this case. However, for many text-classification datasets, it would be typical to see a one-percentage-point increase when using TF-IDF compared to plain binary encoding." ], "metadata": { "id": "qoz77TP4oewU" } }, { "cell_type": "markdown", "source": [ "## Processing words as a sequence: The sequence model approach" ], "metadata": { "id": "ZC7yKzENvV5N" } }, { "cell_type": "markdown", "source": [ "Let’s try out a first sequence model in practice. First, let’s prepare datasets that return integer sequences. In order to keep a manageable input size, **we’ll truncate the inputs after the first 600 words.**\n", "\n", "This is a reasonable choice, since the average review length is 233 words, and only 5% of reviews are longer than 600 words." ], "metadata": { "id": "IC3yN0qevZM2" } }, { "cell_type": "code", "source": [ "max_length = 600\n", "max_tokens = 20000\n", "text_vectorization = layers.TextVectorization(\n", " max_tokens=max_tokens,\n", " output_mode=\"int\",\n", " output_sequence_length=max_length,\n", ")\n", "text_vectorization.adapt(text_only_train_ds)\n", "\n", "int_train_ds = train_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "int_val_ds = val_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)\n", "int_test_ds = test_ds.map(\n", " lambda x, y: (text_vectorization(x), y),\n", " num_parallel_calls=4)" ], "metadata": { "id": "8__1VvdClKay" }, "execution_count": 93, "outputs": [] }, { "cell_type": "markdown", "source": [ "Next, let’s make a model. The simplest way to convert our integer sequences to vector sequences is to one-hot encode the integers (each dimension would represent one possible term in the vocabulary). On top of these one-hot vectors, we’ll add a simple bidirectional LSTM." ], "metadata": { "id": "ezd52NzTvmwl" } }, { "cell_type": "code", "source": [ "inputs = keras.Input(shape=(None,), dtype=\"int64\") # One input is a sequence of integers\n", "embedded = tf.one_hot(inputs, depth=max_tokens) # A 3D tensor of shape [batch size, time steps, embedding size]\n", "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n", "x = layers.Dropout(0.5)(x)\n", "outputs = layers.Dense(1, activation=\"sigmoid\")(x) # Classification layer\n", "model = keras.Model(inputs, outputs)\n", "model.compile(optimizer=\"nadam\",\n", " loss=\"binary_crossentropy\",\n", " metrics=[\"accuracy\"])\n", "model.summary()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jEGBs6wjvs5s", "outputId": "29b2e209-9bff-41ad-8f6d-cd443ffe2480" }, "execution_count": 99, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model_5\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_8 (InputLayer) [(None, None)] 0 \n", " \n", " tf.one_hot_5 (TFOpLambda) (None, None, 20000) 0 \n", " \n", " bidirectional_3 (Bidirectio (None, 64) 5128448 \n", " nal) \n", " \n", " dropout_5 (Dropout) (None, 64) 0 \n", " \n", " dense_15 (Dense) (None, 1) 65 \n", " \n", "=================================================================\n", "Total params: 5,128,513\n", "Trainable params: 5,128,513\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ] }, { "cell_type": "markdown", "source": [ "A first observation: this model will train very slowly, especially compared to the lightweight model of the previous section. This is because our inputs are quite large: each input sample is encoded as a matrix of size `(600, 20000)` (600 words per sample, 20,000 possible words). That’s 12,000,000 floats for a single movie review. Our bidirectional LSTM has a lot of work to do.\n", "\n", "Let's try word embedding. What makes a good word-embedding space depends heavily on your task: the perfect word-embedding space for an English-language movie-review sentiment-analysis model may look different from the perfect embedding space for an English-language legal-document classification model, because the importance of certain semantic relationships varies from task to task. It’s thus reasonable to learn a new embedding space with every new task. Fortunately, backpropagation makes this easy, and Keras makes it even easier. It’s about learning the weights of a layer: the Embedding layer." ], "metadata": { "id": "fUDH3SWRz9u3" } }, { "cell_type": "code", "source": [ "embedding_layer = layers.Embedding(input_dim=max_tokens, output_dim=256)" ], "metadata": { "id": "YYAaa3CNzbWY" }, "execution_count": 102, "outputs": [] }, { "cell_type": "markdown", "source": [ "The Embedding layer is best understood as a dictionary that maps integer indices (which stand for specific words) to dense vectors. The Embedding layer takes as input a rank-2 tensor of integers, of shape `(batch_size,sequence_length)`, where each entry is a sequence of integers. The layer then returns a 3D floating-point tensor of shape` (batch_size, sequence_length, embedding_dimensionality)`.\n", "\n", "When you instantiate an Embedding layer, its weights (its internal dictionary of\n", "token vectors) are initially random, just as with any other layer. During training, these word vectors are gradually adjusted via backpropagation, structuring the space into something the downstream model can exploit. Once fully trained, the embedding space will show a lot of structure—a kind of structure specialized for the specific problem for which you’re training your model. We will talk more about embedding in Lecture 7." ], "metadata": { "id": "aspfIuYU1b83" } }, { "cell_type": "code", "source": [ "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n", "embedded = layers.Embedding(input_dim=max_tokens, output_dim=256)(inputs)\n", "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n", "x = layers.Dropout(0.5)(x)\n", "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n", "model = keras.Model(inputs, outputs)\n", "model.compile(optimizer=\"rmsprop\",\n", " loss=\"binary_crossentropy\",\n", " metrics=[\"accuracy\"])\n", "model.summary()\n", "\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n", "model = keras.models.load_model(\"embeddings_bidir_gru.keras\")\n", "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d1F2Z7fj1UJ-", "outputId": "0d76ca08-5e7c-4951-8d2f-508e0cc0f415" }, "execution_count": 103, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model_6\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_9 (InputLayer) [(None, None)] 0 \n", " \n", " embedding_1 (Embedding) (None, None, 256) 5120000 \n", " \n", " bidirectional_4 (Bidirectio (None, 64) 73984 \n", " nal) \n", " \n", " dropout_6 (Dropout) (None, 64) 0 \n", " \n", " dense_16 (Dense) (None, 1) 65 \n", " \n", "=================================================================\n", "Total params: 5,194,049\n", "Trainable params: 5,194,049\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/10\n", "625/625 [==============================] - 162s 250ms/step - loss: 0.4738 - accuracy: 0.7908 - val_loss: 0.3579 - val_accuracy: 0.8644\n", "Epoch 2/10\n", "625/625 [==============================] - 163s 258ms/step - loss: 0.3071 - accuracy: 0.8917 - val_loss: 0.3044 - val_accuracy: 0.8774\n", "Epoch 3/10\n", "625/625 [==============================] - 242s 385ms/step - loss: 0.2422 - accuracy: 0.9157 - val_loss: 0.3072 - val_accuracy: 0.8870\n", "Epoch 4/10\n", "625/625 [==============================] - 227s 363ms/step - loss: 0.2074 - accuracy: 0.9307 - val_loss: 0.3473 - val_accuracy: 0.8794\n", "Epoch 5/10\n", "625/625 [==============================] - 144s 229ms/step - loss: 0.1751 - accuracy: 0.9432 - val_loss: 0.3301 - val_accuracy: 0.8856\n", "Epoch 6/10\n", "625/625 [==============================] - 199s 316ms/step - loss: 0.1413 - accuracy: 0.9546 - val_loss: 0.4382 - val_accuracy: 0.8706\n", "Epoch 7/10\n", "625/625 [==============================] - 248s 393ms/step - loss: 0.1254 - accuracy: 0.9586 - val_loss: 0.3836 - val_accuracy: 0.8816\n", "Epoch 8/10\n", "625/625 [==============================] - 188s 298ms/step - loss: 0.1065 - accuracy: 0.9660 - val_loss: 0.3809 - val_accuracy: 0.8860\n", "Epoch 9/10\n", "625/625 [==============================] - 198s 314ms/step - loss: 0.0907 - accuracy: 0.9717 - val_loss: 0.4376 - val_accuracy: 0.8440\n", "Epoch 10/10\n", "625/625 [==============================] - 286s 454ms/step - loss: 0.0789 - accuracy: 0.9750 - val_loss: 0.4318 - val_accuracy: 0.8844\n", "782/782 [==============================] - 119s 148ms/step - loss: 0.3496 - accuracy: 0.8573\n", "Test acc: 0.857\n" ] } ] }, { "cell_type": "markdown", "source": [ "We’re still some way off from the results of our basic bigram model. Part of the reason why is simply that the model is looking at slightly less data:\n", "the bigram model processed full reviews, while our sequence model truncates sequences after 600 words. One thing that’s slightly hurting model performance here is that our input sequences are full of zeros. This comes from our use of the `output_sequence_length=max_length` option in `TextVectorization` (with max_length equal to 600): **sentences longer than 600 tokens are truncated to a length of 600 tokens, and sentences shorter than 600 tokens are padded with zeros** at the end so that they can be concatenated together with other sequences to form contiguous batches.\n", "\n", "We’re using a bidirectional RNN: two RNN layers running in parallel, with one processing the tokens in their natural order, and the other processing the same\n", "tokens in reverse. **The RNN that looks at the tokens in their natural order will spend its last iterations seeing only vectors that encode padding—possibly for several hundreds of iterations if the original sentence was short**. The information stored in the internal state of the RNN will gradually fade out as it gets exposed to these meaningless inputs.\n", "\n", "We need some way to tell the RNN that it should skip these iterations. There’s an API for that: *masking*. The Embedding layer is capable of generating a“mask” that corresponds to its input data. This mask is a tensor of ones and zeros (or `True/False` booleans), of shape `(batch_size, sequence_length)`,where the entry `mask[i, t]` indicates where timestep t of sample i should be skipped or not (the timestep will be skipped if `mask[i, t]` is 0 or False, and processed otherwise)." ], "metadata": { "id": "_3dQg0bM2Ks3" } }, { "cell_type": "code", "source": [ "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n", "embedded = layers.Embedding(\n", " input_dim=max_tokens, output_dim=256, mask_zero=True)(inputs) # You can turn it on by passing mask_zero=True\n", "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n", "x = layers.Dropout(0.5)(x)\n", "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n", "model = keras.Model(inputs, outputs)\n", "model.compile(optimizer=\"rmsprop\",\n", " loss=\"binary_crossentropy\",\n", " metrics=[\"accuracy\"])\n", "model.summary()\n", "\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru_with_masking.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n", "model = keras.models.load_model(\"embeddings_bidir_gru_with_masking.keras\")\n", "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "h--MrGck12LD", "outputId": "48d231b8-a68d-4644-c2a3-d9add6b8cd68" }, "execution_count": 104, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model_7\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_10 (InputLayer) [(None, None)] 0 \n", " \n", " embedding_2 (Embedding) (None, None, 256) 5120000 \n", " \n", " bidirectional_5 (Bidirectio (None, 64) 73984 \n", " nal) \n", " \n", " dropout_7 (Dropout) (None, 64) 0 \n", " \n", " dense_17 (Dense) (None, 1) 65 \n", " \n", "=================================================================\n", "Total params: 5,194,049\n", "Trainable params: 5,194,049\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/10\n", "625/625 [==============================] - 187s 280ms/step - loss: 0.4069 - accuracy: 0.8126 - val_loss: 0.2861 - val_accuracy: 0.8832\n", "Epoch 2/10\n", "625/625 [==============================] - 189s 299ms/step - loss: 0.2301 - accuracy: 0.9140 - val_loss: 0.2878 - val_accuracy: 0.8840\n", "Epoch 3/10\n", "625/625 [==============================] - 159s 252ms/step - loss: 0.1745 - accuracy: 0.9373 - val_loss: 0.2828 - val_accuracy: 0.8914\n", "Epoch 4/10\n", "625/625 [==============================] - 178s 282ms/step - loss: 0.1292 - accuracy: 0.9537 - val_loss: 0.3062 - val_accuracy: 0.8916\n", "Epoch 5/10\n", "625/625 [==============================] - 231s 366ms/step - loss: 0.0929 - accuracy: 0.9671 - val_loss: 0.3851 - val_accuracy: 0.8924\n", "Epoch 6/10\n", "625/625 [==============================] - 225s 357ms/step - loss: 0.0725 - accuracy: 0.9752 - val_loss: 0.3541 - val_accuracy: 0.8862\n", "Epoch 7/10\n", "625/625 [==============================] - 206s 326ms/step - loss: 0.0489 - accuracy: 0.9845 - val_loss: 0.6190 - val_accuracy: 0.8332\n", "Epoch 8/10\n", "625/625 [==============================] - 214s 339ms/step - loss: 0.0384 - accuracy: 0.9875 - val_loss: 0.4624 - val_accuracy: 0.8776\n", "Epoch 9/10\n", "625/625 [==============================] - 163s 257ms/step - loss: 0.0260 - accuracy: 0.9912 - val_loss: 0.5290 - val_accuracy: 0.8764\n", "Epoch 10/10\n", "625/625 [==============================] - 184s 291ms/step - loss: 0.0191 - accuracy: 0.9939 - val_loss: 0.7019 - val_accuracy: 0.8642\n", "782/782 [==============================] - 142s 176ms/step - loss: 0.3108 - accuracy: 0.8718\n", "Test acc: 0.872\n" ] } ] }, { "cell_type": "markdown", "source": [ "## The Transformer encoder" ], "metadata": { "id": "hycMo2qy3uO_" } }, { "cell_type": "markdown", "source": [ "The encoder part of transformer can be used for text classification—it’s a very generic module that ingests a sequence and learns to turn it into a more useful representation. Let’s implement a Transformer encoder using Kera subclassing API." ], "metadata": { "id": "ozI7Y4wx6ScJ" } }, { "cell_type": "code", "source": [ "class TransformerEncoder(layers.Layer):\n", " def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n", " super().__init__(**kwargs)\n", " self.embed_dim = embed_dim # Size of the input token vectors\n", " self.dense_dim = dense_dim # Size of the inner dense layer\n", " self.num_heads = num_heads # Number of attention heads\n", " self.attention = layers.MultiHeadAttention(\n", " num_heads=num_heads, key_dim=embed_dim)\n", " self.dense_proj = keras.Sequential(\n", " [layers.Dense(dense_dim, activation=\"relu\"),\n", " layers.Dense(embed_dim),]\n", " )\n", " self.layernorm_1 = layers.LayerNormalization()\n", " self.layernorm_2 = layers.LayerNormalization()\n", " # Computation goes in call().\n", " def call(self, inputs, mask=None): \n", " # The mask that will be generated by the Embedding layer will be 2D, but\n", " # the attention layer expects to be 3D or 4D, so we expand its rank. \n", " if mask is not None:\n", " mask = mask[:, tf.newaxis, :]\n", " attention_output = self.attention(\n", " inputs, inputs, attention_mask=mask)\n", " proj_input = self.layernorm_1(inputs + attention_output)\n", " proj_output = self.dense_proj(proj_input)\n", " return self.layernorm_2(proj_input + proj_output)\n", " # Implement serialization so we can save the model.\n", " def get_config(self):\n", " config = super().get_config()\n", " config.update({\n", " \"embed_dim\": self.embed_dim,\n", " \"num_heads\": self.num_heads,\n", " \"dense_dim\": self.dense_dim,\n", " })\n", " return config" ], "metadata": { "id": "3mjNWl3L5SFK" }, "execution_count": 105, "outputs": [] }, { "cell_type": "markdown", "source": [ "When you write custom layers, make sure to implement the `get_config` method: this enables the layer to be reinstantiated from its config dict, which is useful during model saving and loading.\n", "\n", "To add positional encoding, we’ll do something simpler and more effective: we’ll learn positionembedding vectors the same way we learn to embed word indices. We’ll then proceed to add our position embeddings to the corresponding word embeddings, to obtain a position-aware word embedding. This technique is called **“positional embedding.”** Let’s implement it. **It is noted that vector. \n", "neural networks don’t like very large input values, or discrete input distributions** therefore simply adding a position information as interger is not a good idea." ], "metadata": { "id": "-NEaOFJs7aTW" } }, { "cell_type": "code", "source": [ "class PositionalEmbedding(layers.Layer):\n", " # A downside of position embeddings is that the sequence length needs to be known in advance.\n", " def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n", " super().__init__(**kwargs)\n", " # Prepare an Embedding layer for the token indices.\n", " self.token_embeddings = layers.Embedding(\n", " input_dim=input_dim, output_dim=output_dim)\n", " # And another one for the token positions\n", " self.position_embeddings = layers.Embedding(\n", " input_dim=sequence_length, output_dim=output_dim)\n", " self.sequence_length = sequence_length\n", " self.input_dim = input_dim\n", " self.output_dim = output_dim\n", "\n", " def call(self, inputs):\n", " length = tf.shape(inputs)[-1]\n", " positions = tf.range(start=0, limit=length, delta=1)\n", " embedded_tokens = self.token_embeddings(inputs)\n", " embedded_positions = self.position_embeddings(positions)\n", " # Add both embedding vectors together\n", " return embedded_tokens + embedded_positions\n", " \n", " # Like the Embedding layer, this layer should be able to generate a\n", " # mask so we can ignore padding 0s in the inputs. The compute_mask\n", " # method will called automatically by the framework, and the\n", " # mask will get propagated to the next layer.\n", " def compute_mask(self, inputs, mask=None):\n", " return tf.math.not_equal(inputs, 0)\n", "\n", " def get_config(self):\n", " config = super().get_config()\n", " config.update({\n", " \"output_dim\": self.output_dim,\n", " \"sequence_length\": self.sequence_length,\n", " \"input_dim\": self.input_dim,\n", " })\n", " return config" ], "metadata": { "id": "kaYlKHpS7hgI" }, "execution_count": 106, "outputs": [] }, { "cell_type": "markdown", "source": [ "All you have to do to start taking word order into account is swap the old Embedding layer with our position-aware version." ], "metadata": { "id": "ac1jhOCK-W0q" } }, { "cell_type": "code", "source": [ "vocab_size = 20000\n", "sequence_length = 600\n", "embed_dim = 256\n", "num_heads = 2\n", "dense_dim = 32\n", "\n", "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n", "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n", "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n", "x = layers.GlobalMaxPooling1D()(x)\n", "x = layers.Dropout(0.5)(x)\n", "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n", "model = keras.Model(inputs, outputs)\n", "model.compile(optimizer=\"rmsprop\",\n", " loss=\"binary_crossentropy\",\n", " metrics=[\"accuracy\"])\n", "model.summary()\n", "\n", "callbacks = [\n", " keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\n", " save_best_only=True)\n", "]\n", "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n", "model = keras.models.load_model(\n", " \"full_transformer_encoder.keras\",\n", " custom_objects={\"TransformerEncoder\": TransformerEncoder,\n", " \"PositionalEmbedding\": PositionalEmbedding})\n", "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "hA5Bdn6B-d-i", "outputId": "3c3ad580-9b4b-49cc-acb3-acee89a813dd" }, "execution_count": 107, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Model: \"model_8\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_11 (InputLayer) [(None, None)] 0 \n", " \n", " positional_embedding (Posit (None, None, 256) 5273600 \n", " ionalEmbedding) \n", " \n", " transformer_encoder (Transf (None, None, 256) 543776 \n", " ormerEncoder) \n", " \n", " global_max_pooling1d (Globa (None, 256) 0 \n", " lMaxPooling1D) \n", " \n", " dropout_8 (Dropout) (None, 256) 0 \n", " \n", " dense_20 (Dense) (None, 1) 257 \n", " \n", "=================================================================\n", "Total params: 5,817,633\n", "Trainable params: 5,817,633\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/20\n", "625/625 [==============================] - 112s 174ms/step - loss: 0.4764 - accuracy: 0.7829 - val_loss: 0.2859 - val_accuracy: 0.8824\n", "Epoch 2/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.2302 - accuracy: 0.9129 - val_loss: 0.2564 - val_accuracy: 0.8934\n", "Epoch 3/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.1735 - accuracy: 0.9365 - val_loss: 0.2931 - val_accuracy: 0.8946\n", "Epoch 4/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.1459 - accuracy: 0.9466 - val_loss: 0.3388 - val_accuracy: 0.8904\n", "Epoch 5/20\n", "625/625 [==============================] - 109s 174ms/step - loss: 0.1228 - accuracy: 0.9556 - val_loss: 0.4234 - val_accuracy: 0.8728\n", "Epoch 6/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.1067 - accuracy: 0.9618 - val_loss: 0.2989 - val_accuracy: 0.8886\n", "Epoch 7/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0963 - accuracy: 0.9661 - val_loss: 0.3570 - val_accuracy: 0.8796\n", "Epoch 8/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0869 - accuracy: 0.9699 - val_loss: 0.4418 - val_accuracy: 0.8804\n", "Epoch 9/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0785 - accuracy: 0.9734 - val_loss: 0.3722 - val_accuracy: 0.8790\n", "Epoch 10/20\n", "625/625 [==============================] - 107s 172ms/step - loss: 0.0722 - accuracy: 0.9750 - val_loss: 0.3609 - val_accuracy: 0.8838\n", "Epoch 11/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.0688 - accuracy: 0.9776 - val_loss: 0.4718 - val_accuracy: 0.8834\n", "Epoch 12/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.0612 - accuracy: 0.9803 - val_loss: 0.4938 - val_accuracy: 0.8778\n", "Epoch 13/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0582 - accuracy: 0.9808 - val_loss: 0.4725 - val_accuracy: 0.8712\n", "Epoch 14/20\n", "625/625 [==============================] - 109s 173ms/step - loss: 0.0553 - accuracy: 0.9826 - val_loss: 0.4691 - val_accuracy: 0.8744\n", "Epoch 15/20\n", "625/625 [==============================] - 109s 173ms/step - loss: 0.0480 - accuracy: 0.9844 - val_loss: 0.5810 - val_accuracy: 0.8750\n", "Epoch 16/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0441 - accuracy: 0.9862 - val_loss: 0.5527 - val_accuracy: 0.8716\n", "Epoch 17/20\n", "625/625 [==============================] - 108s 173ms/step - loss: 0.0410 - accuracy: 0.9869 - val_loss: 0.5501 - val_accuracy: 0.8516\n", "Epoch 18/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.0374 - accuracy: 0.9882 - val_loss: 0.5607 - val_accuracy: 0.8714\n", "Epoch 19/20\n", "625/625 [==============================] - 108s 172ms/step - loss: 0.0353 - accuracy: 0.9892 - val_loss: 0.6018 - val_accuracy: 0.8660\n", "Epoch 20/20\n", "625/625 [==============================] - 107s 172ms/step - loss: 0.0282 - accuracy: 0.9916 - val_loss: 0.7235 - val_accuracy: 0.8712\n", "782/782 [==============================] - 49s 62ms/step - loss: 0.2853 - accuracy: 0.8802\n", "Test acc: 0.880\n" ] } ] }, { "cell_type": "code", "source": [ "" ], "metadata": { "id": "9cSp01jOCD_R" }, "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" }, "nav_menu": {}, "toc": { "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 6, "toc_cell": false, "toc_section_display": "block", "toc_window_display": false }, "colab": { "name": "04_Recurrent Neural Networks.ipynb", "provenance": [], "toc_visible": true, "collapsed_sections": [] }, "accelerator": "GPU" }, "nbformat": 4, "nbformat_minor": 0 }